Jul 24 12:19:26.574680 ip-10-0-128-194 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 12:19:27.013175 ip-10-0-128-194 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 12:19:27.013175 ip-10-0-128-194 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 12:19:27.013175 ip-10-0-128-194 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 12:19:27.013175 ip-10-0-128-194 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 12:19:27.013175 ip-10-0-128-194 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 12:19:27.013992 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.013900    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 12:19:27.020152 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020134    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 12:19:27.020152 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020153    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 12:19:27.020225 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020158    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 12:19:27.020225 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020162    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 12:19:27.020225 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020164    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 12:19:27.020225 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020167    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 12:19:27.020225 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020170    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 12:19:27.020225 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020172    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 12:19:27.020225 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020175    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 12:19:27.020225 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020178    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 12:19:27.020225 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020180    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 12:19:27.020225 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020184    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 12:19:27.020225 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020188    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 12:19:27.020225 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020199    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 12:19:27.020225 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020202    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 12:19:27.020225 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020205    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 12:19:27.020225 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020209    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 12:19:27.020225 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020212    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 12:19:27.020225 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020215    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 12:19:27.020225 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020218    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 12:19:27.020225 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020221    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 12:19:27.020686 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020224    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 12:19:27.020686 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020227    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 12:19:27.020686 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020230    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 12:19:27.020686 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020232    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 12:19:27.020686 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020235    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 12:19:27.020686 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020238    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 12:19:27.020686 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020240    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 12:19:27.020686 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020243    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 12:19:27.020686 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020246    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 12:19:27.020686 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020249    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 12:19:27.020686 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020251    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 12:19:27.020686 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020254    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 12:19:27.020686 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020256    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 12:19:27.020686 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020258    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 12:19:27.020686 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020261    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 12:19:27.020686 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020264    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 12:19:27.020686 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020266    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 12:19:27.020686 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020268    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 12:19:27.020686 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020271    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 12:19:27.020686 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020273    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 12:19:27.021184 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020275    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 12:19:27.021184 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020278    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 12:19:27.021184 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020280    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 12:19:27.021184 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020282    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 12:19:27.021184 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020285    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 12:19:27.021184 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020287    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 12:19:27.021184 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020290    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 12:19:27.021184 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020292    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 12:19:27.021184 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020295    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 12:19:27.021184 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020297    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 12:19:27.021184 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020300    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 12:19:27.021184 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020302    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 12:19:27.021184 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020304    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 12:19:27.021184 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020307    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 12:19:27.021184 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020311    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 12:19:27.021184 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020313    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 12:19:27.021184 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020316    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 12:19:27.021184 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020319    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 12:19:27.021184 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020321    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 12:19:27.021184 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020323    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 12:19:27.021687 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020326    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 12:19:27.021687 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020328    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 12:19:27.021687 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020346    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 12:19:27.021687 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020349    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 12:19:27.021687 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020351    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 12:19:27.021687 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020355    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 12:19:27.021687 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020359    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 12:19:27.021687 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020362    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 12:19:27.021687 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020365    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 12:19:27.021687 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020369    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 12:19:27.021687 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020371    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 12:19:27.021687 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020374    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 12:19:27.021687 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020377    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 12:19:27.021687 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020380    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 12:19:27.021687 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020382    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 12:19:27.021687 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020384    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 12:19:27.021687 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020387    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 12:19:27.021687 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020390    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 12:19:27.021687 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020392    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 12:19:27.022136 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020395    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 12:19:27.022136 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020397    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 12:19:27.022136 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020400    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 12:19:27.022136 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020403    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 12:19:27.022136 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020405    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 12:19:27.022136 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020408    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 12:19:27.022136 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020414    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 12:19:27.022136 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020417    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 12:19:27.022136 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020419    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 12:19:27.022136 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020422    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 12:19:27.022136 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020424    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 12:19:27.022136 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020427    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 12:19:27.022136 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020430    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 12:19:27.022136 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020432    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 12:19:27.022136 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020435    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 12:19:27.022136 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020438    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 12:19:27.022136 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020440    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 12:19:27.022136 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020443    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 12:19:27.022136 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.020445    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 12:19:27.022136 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021152    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 12:19:27.022621 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021158    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 12:19:27.022621 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021161    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 12:19:27.022621 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021164    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 12:19:27.022621 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021167    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 12:19:27.022621 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021169    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 12:19:27.022621 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021172    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 12:19:27.022621 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021175    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 12:19:27.022621 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021178    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 12:19:27.022621 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021180    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 12:19:27.022621 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021183    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 12:19:27.022621 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021186    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 12:19:27.022621 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021188    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 12:19:27.022621 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021192    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 12:19:27.022621 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021194    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 12:19:27.022621 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021197    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 12:19:27.022621 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021200    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 12:19:27.022621 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021203    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 12:19:27.022621 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021206    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 12:19:27.022621 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021209    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 12:19:27.022621 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021212    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 12:19:27.023107 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021215    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 12:19:27.023107 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021217    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 12:19:27.023107 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021220    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 12:19:27.023107 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021224    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 12:19:27.023107 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021227    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 12:19:27.023107 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021230    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 12:19:27.023107 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021233    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 12:19:27.023107 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021236    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 12:19:27.023107 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021238    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 12:19:27.023107 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021241    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 12:19:27.023107 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021244    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 12:19:27.023107 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021247    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 12:19:27.023107 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021249    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 12:19:27.023107 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021252    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 12:19:27.023107 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021255    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 12:19:27.023107 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021257    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 12:19:27.023107 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021259    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 12:19:27.023107 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021262    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 12:19:27.023107 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021264    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 12:19:27.023596 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021267    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 12:19:27.023596 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021269    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 12:19:27.023596 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021272    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 12:19:27.023596 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021274    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 12:19:27.023596 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021277    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 12:19:27.023596 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021279    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 12:19:27.023596 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021283    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 12:19:27.023596 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021285    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 12:19:27.023596 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021288    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 12:19:27.023596 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021290    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 12:19:27.023596 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021292    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 12:19:27.023596 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021296    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 12:19:27.023596 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021298    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 12:19:27.023596 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021301    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 12:19:27.023596 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021303    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 12:19:27.023596 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021306    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 12:19:27.023596 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021309    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 12:19:27.023596 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021311    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 12:19:27.023596 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021313    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 12:19:27.024044 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021316    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 12:19:27.024044 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021318    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 12:19:27.024044 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021321    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 12:19:27.024044 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021323    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 12:19:27.024044 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021325    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 12:19:27.024044 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021328    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 12:19:27.024044 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021330    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 12:19:27.024044 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021349    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 12:19:27.024044 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021351    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 12:19:27.024044 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021354    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 12:19:27.024044 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021357    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 12:19:27.024044 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021359    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 12:19:27.024044 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021363    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 12:19:27.024044 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021367    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 12:19:27.024044 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021370    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 12:19:27.024044 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021375    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 12:19:27.024044 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021379    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 12:19:27.024044 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021382    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 12:19:27.024044 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021386    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 12:19:27.024521 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021389    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 12:19:27.024521 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021392    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 12:19:27.024521 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021395    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 12:19:27.024521 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021398    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 12:19:27.024521 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021401    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 12:19:27.024521 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021405    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 12:19:27.024521 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021407    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 12:19:27.024521 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021410    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 12:19:27.024521 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021412    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 12:19:27.024521 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021415    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 12:19:27.024521 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021418    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 12:19:27.024521 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021420    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 12:19:27.024521 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021423    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 12:19:27.024521 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021425    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 12:19:27.024521 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021428    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 12:19:27.024521 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021430    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 12:19:27.024521 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021433    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 12:19:27.024521 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021435    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 12:19:27.024521 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021438    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 12:19:27.024521 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021440    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 12:19:27.025012 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021443    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 12:19:27.025012 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021815    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 12:19:27.025012 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021819    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 12:19:27.025012 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021822    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 12:19:27.025012 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021825    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 12:19:27.025012 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021828    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 12:19:27.025012 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021830    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 12:19:27.025012 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021833    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 12:19:27.025012 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021836    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 12:19:27.025012 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021838    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 12:19:27.025012 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021841    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 12:19:27.025012 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021843    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 12:19:27.025012 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021846    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 12:19:27.025012 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021849    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 12:19:27.025012 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021852    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 12:19:27.025012 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021854    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 12:19:27.025012 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021857    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 12:19:27.025012 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021862    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 12:19:27.025012 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021865    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 12:19:27.025499 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021868    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 12:19:27.025499 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021870    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 12:19:27.025499 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021873    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 12:19:27.025499 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021875    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 12:19:27.025499 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021878    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 12:19:27.025499 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021881    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 12:19:27.025499 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021884    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 12:19:27.025499 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021887    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 12:19:27.025499 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021890    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 12:19:27.025499 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021892    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 12:19:27.025499 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021895    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 12:19:27.025499 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021897    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 12:19:27.025499 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021900    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 12:19:27.025499 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021902    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 12:19:27.025499 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021905    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 12:19:27.025499 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021909    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 12:19:27.025499 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021912    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 12:19:27.025499 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021914    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 12:19:27.025499 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021917    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 12:19:27.025955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021920    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 12:19:27.025955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021922    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 12:19:27.025955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021925    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 12:19:27.025955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021928    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 12:19:27.025955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021930    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 12:19:27.025955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021933    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 12:19:27.025955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021935    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 12:19:27.025955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021938    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 12:19:27.025955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021941    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 12:19:27.025955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021943    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 12:19:27.025955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021946    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 12:19:27.025955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021950    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 12:19:27.025955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021952    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 12:19:27.025955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021954    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 12:19:27.025955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021957    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 12:19:27.025955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021959    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 12:19:27.025955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021962    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 12:19:27.025955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021966    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 12:19:27.025955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021969    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 12:19:27.025955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021972    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 12:19:27.026448 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021975    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 12:19:27.026448 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021978    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 12:19:27.026448 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021980    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 12:19:27.026448 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021983    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 12:19:27.026448 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021985    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 12:19:27.026448 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021988    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 12:19:27.026448 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021991    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 12:19:27.026448 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.021995    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 12:19:27.026448 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022001    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 12:19:27.026448 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022004    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 12:19:27.026448 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022006    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 12:19:27.026448 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022009    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 12:19:27.026448 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022011    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 12:19:27.026448 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022013    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 12:19:27.026448 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022016    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 12:19:27.026448 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022020    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 12:19:27.026448 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022023    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 12:19:27.026448 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022025    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 12:19:27.026448 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022028    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 12:19:27.026448 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022031    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 12:19:27.026954 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022034    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 12:19:27.026954 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022037    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 12:19:27.026954 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022039    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 12:19:27.026954 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022043    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 12:19:27.026954 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022045    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 12:19:27.026954 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022048    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 12:19:27.026954 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022051    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 12:19:27.026954 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022053    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 12:19:27.026954 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022056    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 12:19:27.026954 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022058    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 12:19:27.026954 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022061    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 12:19:27.026954 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022063    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 12:19:27.026954 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022066    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 12:19:27.026954 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022068    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 12:19:27.026954 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022071    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 12:19:27.026954 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022073    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 12:19:27.026954 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022076    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 12:19:27.026954 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022078    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 12:19:27.026954 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022080    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 12:19:27.026954 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022083    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022086    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022089    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022158    2581 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022165    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022171    2581 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022175    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022180    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022184    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022188    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022192    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022195    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022198    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022201    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022204    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022208    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022213    2581 flags.go:64] FLAG: --cgroup-root=""
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022216    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022219    2581 flags.go:64] FLAG: --client-ca-file=""
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022222    2581 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022225    2581 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022236    2581 flags.go:64] FLAG: --cluster-domain=""
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022239    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022242    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 12:19:27.027442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022245    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022248    2581 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022252    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022255    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022258    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022262    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022264    2581 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022267    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022270    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022274    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022277    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022281    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022284    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022287    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022289    2581 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022292    2581 flags.go:64] FLAG: --enable-server="true"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022295    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022299    2581 flags.go:64] FLAG: --event-burst="100"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022302    2581 flags.go:64] FLAG: --event-qps="50"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022305    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022308    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022311    2581 flags.go:64] FLAG: --eviction-hard=""
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022315    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022317    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022323    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 12:19:27.028047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022326    2581 flags.go:64] FLAG: --eviction-soft=""
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022329    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022346    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022349    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022352    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022355    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022358    2581 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022361    2581 flags.go:64] FLAG: --feature-gates=""
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022365    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022368    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022371    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022374    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022378    2581 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022380    2581 flags.go:64] FLAG: --help="false"
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022383    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-128-194.ec2.internal"
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022386    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022390    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022393    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022396    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022399    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022402    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022405    2581 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022408    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 12:19:27.028658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022410    2581 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022413    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022416    2581 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022419    2581 flags.go:64] FLAG: --kube-reserved=""
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022422    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022425    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022428    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022430    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022435    2581 flags.go:64] FLAG: --lock-file=""
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022439    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022442    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022445    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022450    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022453    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022456    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022459    2581 flags.go:64] FLAG: --logging-format="text"
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022462    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022465    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022467    2581 flags.go:64] FLAG: --manifest-url=""
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022470    2581 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022474    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022477    2581 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022481    2581 flags.go:64] FLAG: --max-pods="110"
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022484    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022487    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 12:19:27.029277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022490    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022493    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022496    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022499    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022502    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022508    2581 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022511    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022514    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022517    2581 flags.go:64] FLAG: --pod-cidr=""
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022520    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022525    2581 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022528    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022531    2581 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022533    2581 flags.go:64] FLAG: --port="10250"
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022536    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022539    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dd2b6ee05a7a4283"
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022543    2581 flags.go:64] FLAG: --qos-reserved=""
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022547    2581 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022550    2581 flags.go:64] FLAG: --register-node="true"
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022553    2581 flags.go:64] FLAG: --register-with-taints=""
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022557    2581 flags.go:64] FLAG: --registry-burst="10"
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022559    2581 flags.go:64] FLAG: --registry-qps="5"
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022562    2581 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022565    2581 flags.go:64] FLAG: --reserved-memory=""
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022569    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 12:19:27.029909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022572    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022575    2581 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022577    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022580    2581 flags.go:64] FLAG: --runonce="false"
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022583    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022586    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022588    2581 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022591    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022594    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022597    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022600    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022603    2581 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022606    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022609    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022611    2581 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022614    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022617    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022620    2581 flags.go:64] FLAG: --system-cgroups=""
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022623    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022628    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022631    2581 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022634    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022641    2581 flags.go:64] FLAG: --tls-min-version=""
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022644    2581 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022647    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 12:19:27.030669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022651    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022654    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022656    2581 flags.go:64] FLAG: --v="2"
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022660    2581 flags.go:64] FLAG: --version="false"
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022664    2581 flags.go:64] FLAG: --vmodule=""
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022670    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.022673    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022784    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022788    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022791    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022794    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022797    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022799    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022802    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022805    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022808    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022811    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022813    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022816    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022818    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022821    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 12:19:27.031663 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022823    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 12:19:27.032721 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022826    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 12:19:27.032721 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022828    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 12:19:27.032721 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022830    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 12:19:27.032721 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022833    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 12:19:27.032721 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022835    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 12:19:27.032721 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022838    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 12:19:27.032721 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022840    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 12:19:27.032721 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022843    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 12:19:27.032721 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022846    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 12:19:27.032721 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022849    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 12:19:27.032721 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022851    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 12:19:27.032721 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022854    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 12:19:27.032721 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022857    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 12:19:27.032721 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022859    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 12:19:27.032721 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022862    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 12:19:27.032721 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022865    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 12:19:27.032721 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022867    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 12:19:27.032721 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022870    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 12:19:27.032721 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022872    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 12:19:27.032721 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022875    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 12:19:27.033595 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022877    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 12:19:27.033595 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022880    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 12:19:27.033595 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022884    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 12:19:27.033595 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022887    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 12:19:27.033595 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022890    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 12:19:27.033595 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022893    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 12:19:27.033595 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022895    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 12:19:27.033595 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022898    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 12:19:27.033595 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022900    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 12:19:27.033595 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022903    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 12:19:27.033595 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022905    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 12:19:27.033595 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022907    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 12:19:27.033595 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022910    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 12:19:27.033595 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022912    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 12:19:27.033595 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022915    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 12:19:27.033595 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022917    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 12:19:27.033595 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022919    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 12:19:27.033595 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022922    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 12:19:27.033595 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022926    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 12:19:27.034127 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022929    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 12:19:27.034127 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022932    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 12:19:27.034127 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022935    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 12:19:27.034127 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022937    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 12:19:27.034127 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022940    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 12:19:27.034127 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022944    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 12:19:27.034127 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022946    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 12:19:27.034127 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022949    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 12:19:27.034127 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022953    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 12:19:27.034127 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022955    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 12:19:27.034127 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022958    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 12:19:27.034127 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022960    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 12:19:27.034127 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022962    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 12:19:27.034127 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022965    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 12:19:27.034127 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022967    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 12:19:27.034127 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022970    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 12:19:27.034127 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022972    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 12:19:27.034127 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022974    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 12:19:27.034127 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022977    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 12:19:27.034127 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022979    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 12:19:27.034944 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022982    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 12:19:27.034944 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022984    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 12:19:27.034944 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022987    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 12:19:27.034944 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022989    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 12:19:27.034944 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022993    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 12:19:27.034944 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022996    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 12:19:27.034944 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.022999    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 12:19:27.034944 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.023002    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 12:19:27.034944 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.023004    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 12:19:27.034944 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.023007    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 12:19:27.034944 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.023009    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 12:19:27.034944 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.023012    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 12:19:27.034944 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.023014    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 12:19:27.034944 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.023016    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 12:19:27.034944 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.023019    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 12:19:27.034944 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.023021    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 12:19:27.034944 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.023024    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 12:19:27.034944 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.023026    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 12:19:27.034944 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.023029    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 12:19:27.035754 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.023031    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 12:19:27.035754 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.023035    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 12:19:27.035754 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.023037    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 12:19:27.035754 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.023039    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 12:19:27.035754 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.023042    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 12:19:27.035754 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.023045    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 12:19:27.035754 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.023050    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 12:19:27.035754 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.032027    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 12:19:27.035754 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.032049    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 12:19:27.035754 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.032077    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 12:19:27.035754 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.032090    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 12:19:27.035754 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032180    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 12:19:27.035754 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032189    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 12:19:27.035754 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032194    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 12:19:27.035754 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032198    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 12:19:27.036135 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032203    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 12:19:27.036135 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032208    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 12:19:27.036135 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032213    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 12:19:27.036135 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032217    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 12:19:27.036135 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032221    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 12:19:27.036135 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032226    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 12:19:27.036135 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032230    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 12:19:27.036135 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032234    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 12:19:27.036135 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032238    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 12:19:27.036135 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032242    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 12:19:27.036135 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032247    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 12:19:27.036135 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032251    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 12:19:27.036135 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032255    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 12:19:27.036135 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032259    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 12:19:27.036135 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032264    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 12:19:27.036135 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032268    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 12:19:27.036135 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032272    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 12:19:27.036135 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032277    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 12:19:27.036135 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032281    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 12:19:27.036135 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032285    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 12:19:27.036769 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032290    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 12:19:27.036769 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032294    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 12:19:27.036769 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032298    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 12:19:27.036769 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032303    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 12:19:27.036769 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032308    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 12:19:27.036769 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032313    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 12:19:27.036769 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032317    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 12:19:27.036769 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032322    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 12:19:27.036769 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032326    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 12:19:27.036769 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032330    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 12:19:27.036769 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032355    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 12:19:27.036769 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032359    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 12:19:27.036769 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032364    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 12:19:27.036769 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032368    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 12:19:27.036769 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032372    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 12:19:27.036769 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032376    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 12:19:27.036769 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032381    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 12:19:27.036769 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032385    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 12:19:27.036769 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032389    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 12:19:27.036769 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032393    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 12:19:27.037273 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032398    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 12:19:27.037273 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032402    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 12:19:27.037273 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032406    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 12:19:27.037273 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032410    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 12:19:27.037273 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032414    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 12:19:27.037273 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032418    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 12:19:27.037273 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032422    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 12:19:27.037273 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032429    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 12:19:27.037273 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032435    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 12:19:27.037273 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032440    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 12:19:27.037273 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032446    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 12:19:27.037273 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032452    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 12:19:27.037273 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032456    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 12:19:27.037273 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032461    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 12:19:27.037273 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032466    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 12:19:27.037273 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032470    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 12:19:27.037273 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032475    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 12:19:27.037273 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032479    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 12:19:27.037273 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032482    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 12:19:27.037955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032487    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 12:19:27.037955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032492    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 12:19:27.037955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032498    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 12:19:27.037955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032502    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 12:19:27.037955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032506    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 12:19:27.037955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032510    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 12:19:27.037955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032514    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 12:19:27.037955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032517    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 12:19:27.037955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032521    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 12:19:27.037955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032525    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 12:19:27.037955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032529    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 12:19:27.037955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032533    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 12:19:27.037955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032537    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 12:19:27.037955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032541    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 12:19:27.037955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032544    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 12:19:27.037955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032549    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 12:19:27.037955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032554    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 12:19:27.037955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032557    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 12:19:27.037955 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032561    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 12:19:27.038438 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032565    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 12:19:27.038438 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032569    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 12:19:27.038438 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032573    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 12:19:27.038438 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032577    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 12:19:27.038438 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032581    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 12:19:27.038438 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032585    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 12:19:27.038438 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032590    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 12:19:27.038438 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032594    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 12:19:27.038438 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032597    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 12:19:27.038438 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032601    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 12:19:27.038438 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032607    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 12:19:27.038438 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032611    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 12:19:27.038438 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032615    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 12:19:27.038438 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032620    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 12:19:27.038438 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032624    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 12:19:27.038438 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032628    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 12:19:27.038438 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032632    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 12:19:27.038907 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.032640    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 12:19:27.038907 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032833    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 12:19:27.038907 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032843    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 12:19:27.038907 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032848    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 12:19:27.038907 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032853    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 12:19:27.038907 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032857    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 12:19:27.038907 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032861    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 12:19:27.038907 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032865    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 12:19:27.038907 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032870    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 12:19:27.038907 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032874    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 12:19:27.038907 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032878    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 12:19:27.038907 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032882    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 12:19:27.038907 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032886    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 12:19:27.038907 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032891    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 12:19:27.038907 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032896    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 12:19:27.039286 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032900    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 12:19:27.039286 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032905    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 12:19:27.039286 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032909    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 12:19:27.039286 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032913    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 12:19:27.039286 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032917    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 12:19:27.039286 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032922    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 12:19:27.039286 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032926    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 12:19:27.039286 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032930    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 12:19:27.039286 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032934    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 12:19:27.039286 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032938    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 12:19:27.039286 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032945    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 12:19:27.039286 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032952    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 12:19:27.039286 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032957    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 12:19:27.039286 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032962    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 12:19:27.039286 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032966    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 12:19:27.039286 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032971    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 12:19:27.039286 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032976    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 12:19:27.039286 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032981    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 12:19:27.039286 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032986    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 12:19:27.039780 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032990    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 12:19:27.039780 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032994    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 12:19:27.039780 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.032999    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 12:19:27.039780 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033003    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 12:19:27.039780 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033007    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 12:19:27.039780 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033011    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 12:19:27.039780 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033015    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 12:19:27.039780 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033019    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 12:19:27.039780 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033023    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 12:19:27.039780 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033027    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 12:19:27.039780 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033032    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 12:19:27.039780 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033036    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 12:19:27.039780 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033041    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 12:19:27.039780 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033045    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 12:19:27.039780 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033049    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 12:19:27.039780 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033053    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 12:19:27.039780 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033057    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 12:19:27.039780 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033061    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 12:19:27.039780 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033065    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 12:19:27.039780 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033070    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 12:19:27.040552 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033074    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 12:19:27.040552 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033078    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 12:19:27.040552 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033082    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 12:19:27.040552 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033087    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 12:19:27.040552 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033091    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 12:19:27.040552 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033095    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 12:19:27.040552 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033099    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 12:19:27.040552 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033103    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 12:19:27.040552 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033107    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 12:19:27.040552 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033112    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 12:19:27.040552 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033116    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 12:19:27.040552 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033120    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 12:19:27.040552 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033124    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 12:19:27.040552 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033129    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 12:19:27.040552 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033132    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 12:19:27.040552 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033136    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 12:19:27.040552 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033140    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 12:19:27.040552 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033145    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 12:19:27.040552 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033151    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 12:19:27.040552 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033156    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 12:19:27.041371 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033160    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 12:19:27.041371 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033164    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 12:19:27.041371 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033168    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 12:19:27.041371 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033173    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 12:19:27.041371 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033176    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 12:19:27.041371 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033181    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 12:19:27.041371 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033185    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 12:19:27.041371 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033190    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 12:19:27.041371 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033194    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 12:19:27.041371 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033198    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 12:19:27.041371 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033202    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 12:19:27.041371 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033206    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 12:19:27.041371 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033213    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 12:19:27.041371 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033218    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 12:19:27.041371 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033222    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 12:19:27.041371 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033227    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 12:19:27.041371 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033232    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 12:19:27.041371 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033237    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 12:19:27.041371 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033241    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 12:19:27.042141 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033246    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 12:19:27.042141 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033250    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 12:19:27.042141 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033255    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 12:19:27.042141 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033259    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 12:19:27.042141 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033263    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 12:19:27.042141 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033267    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 12:19:27.042141 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:27.033271    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 12:19:27.042141 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.033279    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 12:19:27.042141 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.033479    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 12:19:27.042141 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.036316    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 12:19:27.042141 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.037863    2581 server.go:1019] "Starting client certificate rotation"
Jul 24 12:19:27.042141 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.037974    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 12:19:27.042141 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.038010    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 12:19:27.042581 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.040562    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 12:19:27.042581 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.042354    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 12:19:27.053749 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.053723    2581 log.go:25] "Validated CRI v1 runtime API"
Jul 24 12:19:27.055850 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.055832    2581 log.go:25] "Validated CRI v1 image API"
Jul 24 12:19:27.056668 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.056642    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 12:19:27.060934 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.060909    2581 fs.go:135] Filesystem UUIDs: map[4d8e6aeb-8bc0-475e-92ce-4a7891e8885c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ae89c669-5e3a-4fe5-8c40-b4de060a77e0:/dev/nvme0n1p3]
Jul 24 12:19:27.061021 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.060933    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 12:19:27.065944 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.065823    2581 manager.go:216] Machine: {Timestamp:2026-07-24 12:19:27.064451488 +0000 UTC m=+0.368536678 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098373 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec243081113004753edc4ad020393450 SystemUUID:ec243081-1130-0475-3edc-4ad020393450 BootID:183ff352-19f8-49b9-b463-f35e870ac6c5 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0b:28:4d:40:af Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0b:28:4d:40:af Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:a1:2d:d7:3f:70 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 12:19:27.065944 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.065932    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 12:19:27.066113 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.066055    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 12:19:27.069189 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.069123    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 12:19:27.069920 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.069193    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-194.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 12:19:27.071867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.071852    2581 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 12:19:27.071867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.071870    2581 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 12:19:27.071987 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.071880    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 12:19:27.072600 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.072589    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 12:19:27.073362 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.073345    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 12:19:27.073769 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.073757    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 12:19:27.075603 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.075592    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 12:19:27.075713 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.075705    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 12:19:27.075746 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.075735    2581 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 12:19:27.075746 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.075745    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 12:19:27.075798 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.075759    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 12:19:27.075798 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.075768    2581 kubelet.go:398] "Adding apiserver pod source"
Jul 24 12:19:27.075798 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.075776    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 12:19:27.076210 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.076199    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 12:19:27.076256 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.076218    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 12:19:27.078508 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.078488    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 12:19:27.080218 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.080203    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 12:19:27.080284 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.080234    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 12:19:27.082133 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.082122    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 12:19:27.082183 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.082139    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 12:19:27.082183 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.082146    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 12:19:27.082183 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.082151    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 12:19:27.082183 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.082159    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 12:19:27.082183 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.082164    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 12:19:27.082183 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.082172    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 12:19:27.082183 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.082177    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 12:19:27.082183 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.082184    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 12:19:27.082406 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.082196    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 12:19:27.082406 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.082204    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 12:19:27.082406 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.082209    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 12:19:27.082871 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.082860    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 12:19:27.082871 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.082870    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 12:19:27.085351 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.085313    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-194.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 12:19:27.085592 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.085570    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 12:19:27.085638 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.085581    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 12:19:27.086504 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.086491    2581 server.go:1268] "Started kubelet"
Jul 24 12:19:27.086628 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.086579    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 12:19:27.086731 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.086650    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 12:19:27.086731 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.086588    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 12:19:27.087549 ip-10-0-128-194 systemd[1]: Started Kubernetes Kubelet.
Jul 24 12:19:27.088368 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.088259    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 12:19:27.089011 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.088992    2581 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 12:19:27.093211 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.093194    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 12:19:27.095283 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.095264    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 12:19:27.095395 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.095270    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 12:19:27.095515 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.095491    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-194.ec2.internal\" not found"
Jul 24 12:19:27.095824 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.095804    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 12:19:27.095902 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.095885    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 12:19:27.095902 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.095901    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 12:19:27.096001 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.095990    2581 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 12:19:27.096001 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.095999    2581 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 12:19:27.096207 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.096181    2581 factory.go:55] Registering systemd factory
Jul 24 12:19:27.096207 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.096201    2581 factory.go:223] Registration of the systemd container factory successfully
Jul 24 12:19:27.096483 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.096467    2581 factory.go:153] Registering CRI-O factory
Jul 24 12:19:27.096483 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.096483    2581 factory.go:223] Registration of the crio container factory successfully
Jul 24 12:19:27.096591 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.096526    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 12:19:27.096591 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.096544    2581 factory.go:103] Registering Raw factory
Jul 24 12:19:27.096591 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.096559    2581 manager.go:1196] Started watching for new ooms in manager
Jul 24 12:19:27.096960 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.095982    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-194.ec2.internal.18c5390fbd5ba966  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-194.ec2.internal,UID:ip-10-0-128-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-194.ec2.internal,},FirstTimestamp:2026-07-24 12:19:27.086463334 +0000 UTC m=+0.390548524,LastTimestamp:2026-07-24 12:19:27.086463334 +0000 UTC m=+0.390548524,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-194.ec2.internal,}"
Jul 24 12:19:27.097275 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.097263    2581 manager.go:349] Starting recovery of all containers
Jul 24 12:19:27.108203 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.108158    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 24 12:19:27.108398 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.108368    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 24 12:19:27.109935 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.109823    2581 manager.go:354] Recovery completed
Jul 24 12:19:27.111439 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.111420    2581 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 24 12:19:27.114531 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.114518    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:19:27.116538 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.116518    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:19:27.116600 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.116559    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:19:27.116600 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.116573    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:19:27.117116 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.117100    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 12:19:27.117116 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.117116    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 12:19:27.117295 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.117135    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 12:19:27.117295 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.117116    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:19:27.117841 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.117825    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:19:27.117899 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.117852    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:19:27.117899 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.117862    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:19:27.118743 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.118729    2581 policy_none.go:49] "None policy: Start"
Jul 24 12:19:27.118808 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.118746    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 12:19:27.118808 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.118756    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 12:19:27.119701 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.119690    2581 policy_none.go:47] "Start"
Jul 24 12:19:27.127959 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.127887    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-194.ec2.internal.18c5390fbf2699bb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-194.ec2.internal,UID:ip-10-0-128-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-194.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-194.ec2.internal,},FirstTimestamp:2026-07-24 12:19:27.116540347 +0000 UTC m=+0.420625543,LastTimestamp:2026-07-24 12:19:27.116540347 +0000 UTC m=+0.420625543,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-194.ec2.internal,}"
Jul 24 12:19:27.134798 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.134720    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-194.ec2.internal.18c5390fbf270144  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-194.ec2.internal,UID:ip-10-0-128-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-194.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-194.ec2.internal,},FirstTimestamp:2026-07-24 12:19:27.116566852 +0000 UTC m=+0.420652042,LastTimestamp:2026-07-24 12:19:27.116566852 +0000 UTC m=+0.420652042,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-194.ec2.internal,}"
Jul 24 12:19:27.136754 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.136691    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-194.ec2.internal.18c5390fbf272adc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-194.ec2.internal,UID:ip-10-0-128-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-194.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-194.ec2.internal,},FirstTimestamp:2026-07-24 12:19:27.1165775 +0000 UTC m=+0.420662692,LastTimestamp:2026-07-24 12:19:27.1165775 +0000 UTC m=+0.420662692,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-194.ec2.internal,}"
Jul 24 12:19:27.142294 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.142276    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vkcmh"
Jul 24 12:19:27.145604 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.145538    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-194.ec2.internal.18c5390fbf2699bb\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-194.ec2.internal.18c5390fbf2699bb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-194.ec2.internal,UID:ip-10-0-128-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-194.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-194.ec2.internal,},FirstTimestamp:2026-07-24 12:19:27.116540347 +0000 UTC m=+0.420625543,LastTimestamp:2026-07-24 12:19:27.117839547 +0000 UTC m=+0.421924738,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-194.ec2.internal,}"
Jul 24 12:19:27.151417 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.151400    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vkcmh"
Jul 24 12:19:27.155950 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.155881    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-194.ec2.internal.18c5390fbf270144\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-194.ec2.internal.18c5390fbf270144  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-194.ec2.internal,UID:ip-10-0-128-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-194.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-194.ec2.internal,},FirstTimestamp:2026-07-24 12:19:27.116566852 +0000 UTC m=+0.420652042,LastTimestamp:2026-07-24 12:19:27.11785617 +0000 UTC m=+0.421941360,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-194.ec2.internal,}"
Jul 24 12:19:27.156033 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.156020    2581 manager.go:337] "Starting Device Plugin manager"
Jul 24 12:19:27.156077 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.156066    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 12:19:27.156119 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.156079    2581 server.go:87] "Starting device plugin registration server"
Jul 24 12:19:27.156392 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.156379    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 12:19:27.156466 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.156397    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 12:19:27.156520 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.156506    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 12:19:27.156605 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.156585    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 12:19:27.156605 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.156599    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 12:19:27.157189 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.157164    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 12:19:27.157266 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.157252    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-194.ec2.internal\" not found"
Jul 24 12:19:27.256525 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.256488    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:19:27.257728 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.257710    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:19:27.257799 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.257745    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:19:27.257799 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.257756    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:19:27.257799 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.257789    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-194.ec2.internal"
Jul 24 12:19:27.267087 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.267035    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-194.ec2.internal"
Jul 24 12:19:27.267087 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.267067    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-194.ec2.internal\": node \"ip-10-0-128-194.ec2.internal\" not found"
Jul 24 12:19:27.288421 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.288385    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-194.ec2.internal\" not found"
Jul 24 12:19:27.288566 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.288442    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 12:19:27.289993 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.289972    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 12:19:27.290107 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.290012    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 12:19:27.290107 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.290034    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 12:19:27.290107 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.290068    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 12:19:27.292403 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.292378    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 12:19:27.388564 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.388513    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-194.ec2.internal\" not found"
Jul 24 12:19:27.390673 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.390651    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-194.ec2.internal"]
Jul 24 12:19:27.390764 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.390753    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:19:27.392883 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.392865    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:19:27.392985 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.392897    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:19:27.392985 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.392906    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:19:27.394141 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.394129    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:19:27.394255 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.394240    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal"
Jul 24 12:19:27.394296 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.394269    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:19:27.394937 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.394919    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:19:27.394937 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.394937    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:19:27.395061 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.394949    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:19:27.395061 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.394956    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:19:27.395061 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.394963    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:19:27.395193 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.394965    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:19:27.396396 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.396381    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-194.ec2.internal"
Jul 24 12:19:27.396466 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.396406    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 12:19:27.397025 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.397006    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3643e41074a2c5417adb5993a4721adf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal\" (UID: \"3643e41074a2c5417adb5993a4721adf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal"
Jul 24 12:19:27.397125 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.397045    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3643e41074a2c5417adb5993a4721adf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal\" (UID: \"3643e41074a2c5417adb5993a4721adf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal"
Jul 24 12:19:27.397125 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.397064    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/69ea8b15ea96888b6c85fb2084a0817d-config\") pod \"kube-apiserver-proxy-ip-10-0-128-194.ec2.internal\" (UID: \"69ea8b15ea96888b6c85fb2084a0817d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-194.ec2.internal"
Jul 24 12:19:27.397125 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.397077    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 12:19:27.397125 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.397102    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 12:19:27.397125 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.397116    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeHasSufficientPID"
Jul 24 12:19:27.422973 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.422948    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-194.ec2.internal\" not found" node="ip-10-0-128-194.ec2.internal"
Jul 24 12:19:27.427294 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.427278    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-194.ec2.internal\" not found" node="ip-10-0-128-194.ec2.internal"
Jul 24 12:19:27.488774 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.488748    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-194.ec2.internal\" not found"
Jul 24 12:19:27.497761 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.497736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3643e41074a2c5417adb5993a4721adf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal\" (UID: \"3643e41074a2c5417adb5993a4721adf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal"
Jul 24 12:19:27.497822 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.497766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3643e41074a2c5417adb5993a4721adf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal\" (UID: \"3643e41074a2c5417adb5993a4721adf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal"
Jul 24 12:19:27.497822 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.497782    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/69ea8b15ea96888b6c85fb2084a0817d-config\") pod \"kube-apiserver-proxy-ip-10-0-128-194.ec2.internal\" (UID: \"69ea8b15ea96888b6c85fb2084a0817d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-194.ec2.internal"
Jul 24 12:19:27.497822 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.497808    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/69ea8b15ea96888b6c85fb2084a0817d-config\") pod \"kube-apiserver-proxy-ip-10-0-128-194.ec2.internal\" (UID: \"69ea8b15ea96888b6c85fb2084a0817d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-194.ec2.internal"
Jul 24 12:19:27.497934 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.497824    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3643e41074a2c5417adb5993a4721adf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal\" (UID: \"3643e41074a2c5417adb5993a4721adf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal"
Jul 24 12:19:27.497934 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.497848    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3643e41074a2c5417adb5993a4721adf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal\" (UID: \"3643e41074a2c5417adb5993a4721adf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal"
Jul 24 12:19:27.589207 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.589116    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-194.ec2.internal\" not found"
Jul 24 12:19:27.690061 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.690024    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-194.ec2.internal\" not found"
Jul 24 12:19:27.726530 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.726497    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal"
Jul 24 12:19:27.730429 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:27.730177    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-194.ec2.internal"
Jul 24 12:19:27.790697 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.790660    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-194.ec2.internal\" not found"
Jul 24 12:19:27.891310 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:27.891231    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-194.ec2.internal\" not found"
Jul 24 12:19:29.340883 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:29.340826    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-194.ec2.internal\" not found"
Jul 24 12:19:29.386427 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:29.386392    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 12:19:29.386885 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:29.386545    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 12:19:29.441025 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:29.440991    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-194.ec2.internal\" not found"
Jul 24 12:19:29.504093 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:29.503996    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 12:14:28 +0000 UTC" deadline="2028-03-10 12:43:42.01679388 +0000 UTC"
Jul 24 12:19:29.504093 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:29.504043    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14280h24m12.512754412s"
Jul 24 12:19:29.541332 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:29.541292    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-194.ec2.internal\" not found"
Jul 24 12:19:29.579193 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:29.579161    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 12:19:29.591447 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:29.591400    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 12:19:29.618604 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:29.618567    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3643e41074a2c5417adb5993a4721adf.slice/crio-9af5f8728118ffdf52021300018f7643ad12f5d8a6550531eeb247c560c6820c WatchSource:0}: Error finding container 9af5f8728118ffdf52021300018f7643ad12f5d8a6550531eeb247c560c6820c: Status 404 returned error can't find the container with id 9af5f8728118ffdf52021300018f7643ad12f5d8a6550531eeb247c560c6820c
Jul 24 12:19:29.618884 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:29.618861    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ea8b15ea96888b6c85fb2084a0817d.slice/crio-744c76b3b8a235a25cb995d91d2d48139a0de440d6fcc39721e9e973b7065c46 WatchSource:0}: Error finding container 744c76b3b8a235a25cb995d91d2d48139a0de440d6fcc39721e9e973b7065c46: Status 404 returned error can't find the container with id 744c76b3b8a235a25cb995d91d2d48139a0de440d6fcc39721e9e973b7065c46
Jul 24 12:19:29.625133 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:29.625115    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 12:19:29.643089 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:29.643037    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-194.ec2.internal" event={"ID":"69ea8b15ea96888b6c85fb2084a0817d","Type":"ContainerStarted","Data":"744c76b3b8a235a25cb995d91d2d48139a0de440d6fcc39721e9e973b7065c46"}
Jul 24 12:19:29.643994 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:29.643973    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal" event={"ID":"3643e41074a2c5417adb5993a4721adf","Type":"ContainerStarted","Data":"9af5f8728118ffdf52021300018f7643ad12f5d8a6550531eeb247c560c6820c"}
Jul 24 12:19:29.645077 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:29.645060    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal"
Jul 24 12:19:29.659849 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:29.659828    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 12:19:29.660881 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:29.660868    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-194.ec2.internal"
Jul 24 12:19:29.670447 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:29.670411    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 12:19:29.874017 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:29.873983    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 12:19:30.426088 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.426057    2581 apiserver.go:52] "Watching apiserver"
Jul 24 12:19:30.432076 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.432052    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 12:19:30.434116 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.434056    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-tzz5k","kube-system/global-pull-secret-syncer-qrzvj","kube-system/konnectivity-agent-hrsjh","kube-system/kube-apiserver-proxy-ip-10-0-128-194.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z","openshift-dns/node-resolver-jmmcs","openshift-multus/multus-additional-cni-plugins-bk4w6","openshift-multus/multus-jzmtg","openshift-network-operator/iptables-alerter-k6gfj","openshift-cluster-node-tuning-operator/tuned-tvdgf","openshift-image-registry/node-ca-sj8p9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal","openshift-multus/network-metrics-daemon-wtb2z","openshift-network-diagnostics/network-check-target-8294s"]
Jul 24 12:19:30.435948 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.435929    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.438617 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.437939    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:30.438617 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:30.438029    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtb2z" podUID="bda12f0d-5325-4258-a601-381a0751907a"
Jul 24 12:19:30.438617 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.438451    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-99bkf\""
Jul 24 12:19:30.439386 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.439005    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hrsjh"
Jul 24 12:19:30.439386 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.439125    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 12:19:30.440108 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.440092    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:30.440187 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:30.440159    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8294s" podUID="edc2499b-10da-416b-831d-52f00cbd026e"
Jul 24 12:19:30.444626 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.443649    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.444626 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.443737    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 12:19:30.444626 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.443870    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 12:19:30.444626 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.444039    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-g528b\""
Jul 24 12:19:30.444626 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.444141    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 12:19:30.444626 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.444175    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jmmcs"
Jul 24 12:19:30.444626 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.444488    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 12:19:30.446914 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.446497    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.448839 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.448825    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 12:19:30.448998 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.448981    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ghsb5\""
Jul 24 12:19:30.449101 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.449092    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 12:19:30.449225 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.449152    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 12:19:30.449225 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.449194    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 12:19:30.449366 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.449345    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 12:19:30.449439 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.449362    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 12:19:30.449654 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.449639    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pmrxk\""
Jul 24 12:19:30.450028 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.450009    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.450100 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.450044    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 12:19:30.452208 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.450011    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 12:19:30.452208 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.450499    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 12:19:30.452208 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.450605    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ww6pg\""
Jul 24 12:19:30.452208 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.451193    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:30.452208 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:30.451256    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrzvj" podUID="d98f89d0-11ef-4c6b-b973-64267283f1da"
Jul 24 12:19:30.452906 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.452889    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.454339 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.454303    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sj8p9"
Jul 24 12:19:30.456498 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.456480    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 12:19:30.457109 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.457087    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tbqh6\""
Jul 24 12:19:30.457405 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.457390    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 12:19:30.457533 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.457466    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 12:19:30.457588 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.457566    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 12:19:30.457738 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.457721    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 12:19:30.457846 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.457830    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 12:19:30.457990 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.457973    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 12:19:30.458049 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458016    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 12:19:30.458345 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq2r9\" (UniqueName: \"kubernetes.io/projected/a0e54908-d259-4a41-b758-eb167ca6239c-kube-api-access-zq2r9\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.458409 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458345    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bvjfd\""
Jul 24 12:19:30.458409 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458358    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lvqp\" (UniqueName: \"kubernetes.io/projected/bda12f0d-5325-4258-a601-381a0751907a-kube-api-access-8lvqp\") pod \"network-metrics-daemon-wtb2z\" (UID: \"bda12f0d-5325-4258-a601-381a0751907a\") " pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:30.458409 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/138735f1-3e23-48e4-88a5-7992d2987d07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.458569 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458450    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qjk9\" (UniqueName: \"kubernetes.io/projected/138735f1-3e23-48e4-88a5-7992d2987d07-kube-api-access-7qjk9\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.458569 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458477    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-os-release\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.458569 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458492    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-multus-socket-dir-parent\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.458569 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458515    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-host-var-lib-kubelet\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.458569 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458548    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/138735f1-3e23-48e4-88a5-7992d2987d07-registration-dir\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.458790 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/138735f1-3e23-48e4-88a5-7992d2987d07-device-dir\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.458790 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458641    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a576aeb-3eff-4419-9bee-f606af167d49-cni-binary-copy\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.458790 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458710    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-multus-conf-dir\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.458790 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458747    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-etc-systemd\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.458790 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1dc4198c-786b-4dec-abb0-0838e3fab08a-agent-certs\") pod \"konnectivity-agent-hrsjh\" (UID: \"1dc4198c-786b-4dec-abb0-0838e3fab08a\") " pod="kube-system/konnectivity-agent-hrsjh"
Jul 24 12:19:30.459017 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/138735f1-3e23-48e4-88a5-7992d2987d07-socket-dir\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.459017 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458877    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/138735f1-3e23-48e4-88a5-7992d2987d07-etc-selinux\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.459017 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458911    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/138735f1-3e23-48e4-88a5-7992d2987d07-sys-fs\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.459017 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-multus-cni-dir\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.459017 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-etc-sysconfig\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.459017 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.458984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0e54908-d259-4a41-b758-eb167ca6239c-tmp\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.459017 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459010    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0385bad7-43a0-4ee3-b116-0e89d51f30f7-hosts-file\") pod \"node-resolver-jmmcs\" (UID: \"0385bad7-43a0-4ee3-b116-0e89d51f30f7\") " pod="openshift-dns/node-resolver-jmmcs"
Jul 24 12:19:30.459330 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm2k2\" (UniqueName: \"kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2\") pod \"network-check-target-8294s\" (UID: \"edc2499b-10da-416b-831d-52f00cbd026e\") " pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:30.459330 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-etc-modprobe-d\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.459330 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459163    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1dc4198c-786b-4dec-abb0-0838e3fab08a-konnectivity-ca\") pod \"konnectivity-agent-hrsjh\" (UID: \"1dc4198c-786b-4dec-abb0-0838e3fab08a\") " pod="kube-system/konnectivity-agent-hrsjh"
Jul 24 12:19:30.459330 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459216    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0385bad7-43a0-4ee3-b116-0e89d51f30f7-tmp-dir\") pod \"node-resolver-jmmcs\" (UID: \"0385bad7-43a0-4ee3-b116-0e89d51f30f7\") " pod="openshift-dns/node-resolver-jmmcs"
Jul 24 12:19:30.459330 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459259    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-cnibin\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.459330 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-host-run-k8s-cni-cncf-io\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.459672 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a576aeb-3eff-4419-9bee-f606af167d49-multus-daemon-config\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.459672 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-lib-modules\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.459672 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459367    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 12:19:30.459672 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-var-lib-kubelet\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.459672 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459517    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs\") pod \"network-metrics-daemon-wtb2z\" (UID: \"bda12f0d-5325-4258-a601-381a0751907a\") " pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:30.459672 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-host-run-netns\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.459672 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459579    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-hostroot\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.459672 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459600    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g9tk\" (UniqueName: \"kubernetes.io/projected/3a576aeb-3eff-4419-9bee-f606af167d49-kube-api-access-4g9tk\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.459672 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459624    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-etc-kubernetes\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.459672 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459647    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-sys\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.460099 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459685    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-host\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.460099 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-system-cni-dir\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.460099 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-host-var-lib-cni-bin\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.460099 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459819    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-etc-kubernetes\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.460099 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459851    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-etc-sysctl-conf\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.460099 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459892    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r7dz\" (UniqueName: \"kubernetes.io/projected/0385bad7-43a0-4ee3-b116-0e89d51f30f7-kube-api-access-9r7dz\") pod \"node-resolver-jmmcs\" (UID: \"0385bad7-43a0-4ee3-b116-0e89d51f30f7\") " pod="openshift-dns/node-resolver-jmmcs"
Jul 24 12:19:30.460099 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459932    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-host-var-lib-cni-multus\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.460099 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.459963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-host-run-multus-certs\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.460099 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.460007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-etc-sysctl-d\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.460099 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.460044    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-run\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.460099 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.460087    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a0e54908-d259-4a41-b758-eb167ca6239c-etc-tuned\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.460939 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.460920    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k6gfj"
Jul 24 12:19:30.461387 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.461365    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 12:19:30.462854 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.462837    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 12:19:30.463140 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.463123    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 12:19:30.463340 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.463327    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zctxn\""
Jul 24 12:19:30.463955 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.463828    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qldps\""
Jul 24 12:19:30.464279 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.464047    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 12:19:30.464279 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.464078    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 12:19:30.464279 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.464111    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 12:19:30.489489 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.489339    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8gkvg"
Jul 24 12:19:30.498694 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.498666    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8gkvg"
Jul 24 12:19:30.546222 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.546199    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 12:19:30.561224 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561174    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1dc4198c-786b-4dec-abb0-0838e3fab08a-konnectivity-ca\") pod \"konnectivity-agent-hrsjh\" (UID: \"1dc4198c-786b-4dec-abb0-0838e3fab08a\") " pod="kube-system/konnectivity-agent-hrsjh"
Jul 24 12:19:30.561224 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561208    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-cnibin\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.561474 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a576aeb-3eff-4419-9bee-f606af167d49-multus-daemon-config\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.561474 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561331    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-cnibin\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.561474 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561358    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.561474 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561392    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-cni-bin\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.561474 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561440    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4zb8\" (UniqueName: \"kubernetes.io/projected/45d555da-de43-4a10-85d3-b24a77d99118-kube-api-access-l4zb8\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.561474 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs\") pod \"network-metrics-daemon-wtb2z\" (UID: \"bda12f0d-5325-4258-a601-381a0751907a\") " pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:30.561867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561501    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-host-run-netns\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.561867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561523    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-hostroot\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.561867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g9tk\" (UniqueName: \"kubernetes.io/projected/3a576aeb-3eff-4419-9bee-f606af167d49-kube-api-access-4g9tk\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.561867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561592    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x25gj\" (UniqueName: \"kubernetes.io/projected/c74cea6b-83f7-4615-918a-d27808ed55ee-kube-api-access-x25gj\") pod \"node-ca-sj8p9\" (UID: \"c74cea6b-83f7-4615-918a-d27808ed55ee\") " pod="openshift-image-registry/node-ca-sj8p9"
Jul 24 12:19:30.561867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561616    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.561867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561643    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d98f89d0-11ef-4c6b-b973-64267283f1da-dbus\") pod \"global-pull-secret-syncer-qrzvj\" (UID: \"d98f89d0-11ef-4c6b-b973-64267283f1da\") " pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:30.561867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-host-var-lib-cni-bin\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.561867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-host-run-netns\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.561867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-etc-kubernetes\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.561867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561716    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgrnl\" (UniqueName: \"kubernetes.io/projected/1659a0ea-58c5-41f5-b8a6-b59ac3ae0ef3-kube-api-access-zgrnl\") pod \"iptables-alerter-k6gfj\" (UID: \"1659a0ea-58c5-41f5-b8a6-b59ac3ae0ef3\") " pod="openshift-network-operator/iptables-alerter-k6gfj"
Jul 24 12:19:30.561867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561743    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-cnibin\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.561867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561777    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d98f89d0-11ef-4c6b-b973-64267283f1da-kubelet-config\") pod \"global-pull-secret-syncer-qrzvj\" (UID: \"d98f89d0-11ef-4c6b-b973-64267283f1da\") " pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:30.561867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561819    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-slash\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.561867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561841    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-run-ovn-kubernetes\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.561867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561857    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a576aeb-3eff-4419-9bee-f606af167d49-multus-daemon-config\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.562593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561865    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r7dz\" (UniqueName: \"kubernetes.io/projected/0385bad7-43a0-4ee3-b116-0e89d51f30f7-kube-api-access-9r7dz\") pod \"node-resolver-jmmcs\" (UID: \"0385bad7-43a0-4ee3-b116-0e89d51f30f7\") " pod="openshift-dns/node-resolver-jmmcs"
Jul 24 12:19:30.562593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561914    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-host-var-lib-cni-multus\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.562593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561920    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-hostroot\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.562593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-host-run-multus-certs\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.562593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561944    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1dc4198c-786b-4dec-abb0-0838e3fab08a-konnectivity-ca\") pod \"konnectivity-agent-hrsjh\" (UID: \"1dc4198c-786b-4dec-abb0-0838e3fab08a\") " pod="kube-system/konnectivity-agent-hrsjh"
Jul 24 12:19:30.562593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.561964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-etc-sysctl-d\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.562593 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:30.561621    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:19:30.562593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562020    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-etc-kubernetes\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.562593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq2r9\" (UniqueName: \"kubernetes.io/projected/a0e54908-d259-4a41-b758-eb167ca6239c-kube-api-access-zq2r9\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.562593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562082    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-var-lib-openvswitch\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.562593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lvqp\" (UniqueName: \"kubernetes.io/projected/bda12f0d-5325-4258-a601-381a0751907a-kube-api-access-8lvqp\") pod \"network-metrics-daemon-wtb2z\" (UID: \"bda12f0d-5325-4258-a601-381a0751907a\") " pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:30.562593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/138735f1-3e23-48e4-88a5-7992d2987d07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.562593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qjk9\" (UniqueName: \"kubernetes.io/projected/138735f1-3e23-48e4-88a5-7992d2987d07-kube-api-access-7qjk9\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.562593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-os-release\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.562593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-multus-socket-dir-parent\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.562593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562280    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c74cea6b-83f7-4615-918a-d27808ed55ee-host\") pod \"node-ca-sj8p9\" (UID: \"c74cea6b-83f7-4615-918a-d27808ed55ee\") " pod="openshift-image-registry/node-ca-sj8p9"
Jul 24 12:19:30.562593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562304    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-os-release\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.562593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562328    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-run-netns\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.563362 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/138735f1-3e23-48e4-88a5-7992d2987d07-registration-dir\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.563362 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/138735f1-3e23-48e4-88a5-7992d2987d07-device-dir\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.563362 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a576aeb-3eff-4419-9bee-f606af167d49-cni-binary-copy\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.563362 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-multus-conf-dir\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.563362 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562476    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-run-systemd\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.563362 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562499    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-run-openvswitch\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.563362 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-cni-netd\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.563362 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/138735f1-3e23-48e4-88a5-7992d2987d07-socket-dir\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.563362 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/138735f1-3e23-48e4-88a5-7992d2987d07-etc-selinux\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.563362 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/138735f1-3e23-48e4-88a5-7992d2987d07-sys-fs\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.563362 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.563362 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562712    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0385bad7-43a0-4ee3-b116-0e89d51f30f7-hosts-file\") pod \"node-resolver-jmmcs\" (UID: \"0385bad7-43a0-4ee3-b116-0e89d51f30f7\") " pod="openshift-dns/node-resolver-jmmcs"
Jul 24 12:19:30.563362 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562730    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm2k2\" (UniqueName: \"kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2\") pod \"network-check-target-8294s\" (UID: \"edc2499b-10da-416b-831d-52f00cbd026e\") " pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:30.563362 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-etc-modprobe-d\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.563362 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562775    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-system-cni-dir\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.563362 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0385bad7-43a0-4ee3-b116-0e89d51f30f7-tmp-dir\") pod \"node-resolver-jmmcs\" (UID: \"0385bad7-43a0-4ee3-b116-0e89d51f30f7\") " pod="openshift-dns/node-resolver-jmmcs"
Jul 24 12:19:30.563362 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-host-run-k8s-cni-cncf-io\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.563994 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562829    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-lib-modules\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.563994 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562848    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-var-lib-kubelet\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.563994 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562864    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45d555da-de43-4a10-85d3-b24a77d99118-env-overrides\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.563994 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-etc-kubernetes\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.563994 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-sys\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.563994 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-host\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.563994 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562935    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.563994 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562955    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-system-cni-dir\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.563994 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.562978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-etc-sysctl-conf\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.563994 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563006    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c74cea6b-83f7-4615-918a-d27808ed55ee-serviceca\") pod \"node-ca-sj8p9\" (UID: \"c74cea6b-83f7-4615-918a-d27808ed55ee\") " pod="openshift-image-registry/node-ca-sj8p9"
Jul 24 12:19:30.563994 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563031    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-run\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.563994 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563055    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a0e54908-d259-4a41-b758-eb167ca6239c-etc-tuned\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.563994 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563081    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-etc-openvswitch\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.563994 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563106    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-run-ovn\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.563994 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563131    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45d555da-de43-4a10-85d3-b24a77d99118-ovn-node-metrics-cert\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.563994 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-host-var-lib-kubelet\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.563994 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563223    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1659a0ea-58c5-41f5-b8a6-b59ac3ae0ef3-iptables-alerter-script\") pod \"iptables-alerter-k6gfj\" (UID: \"1659a0ea-58c5-41f5-b8a6-b59ac3ae0ef3\") " pod="openshift-network-operator/iptables-alerter-k6gfj"
Jul 24 12:19:30.564485 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv64b\" (UniqueName: \"kubernetes.io/projected/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-kube-api-access-mv64b\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.564485 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563343    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/138735f1-3e23-48e4-88a5-7992d2987d07-socket-dir\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.564485 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563385    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-system-cni-dir\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.564485 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563404    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-sys\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.564485 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563432    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-host-run-k8s-cni-cncf-io\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.564485 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563467    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/138735f1-3e23-48e4-88a5-7992d2987d07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.564485 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563481    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-etc-sysctl-conf\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.564485 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/138735f1-3e23-48e4-88a5-7992d2987d07-sys-fs\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.564485 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563517    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/138735f1-3e23-48e4-88a5-7992d2987d07-etc-selinux\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.564485 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563553    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-multus-socket-dir-parent\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.564485 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-lib-modules\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.564485 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563576    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0385bad7-43a0-4ee3-b116-0e89d51f30f7-hosts-file\") pod \"node-resolver-jmmcs\" (UID: \"0385bad7-43a0-4ee3-b116-0e89d51f30f7\") " pod="openshift-dns/node-resolver-jmmcs"
Jul 24 12:19:30.564485 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563607    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-systemd-units\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.564485 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563631    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-var-lib-kubelet\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.564485 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:30.563649    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs podName:bda12f0d-5325-4258-a601-381a0751907a nodeName:}" failed. No retries permitted until 2026-07-24 12:19:31.063610972 +0000 UTC m=+3.018615735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs") pod "network-metrics-daemon-wtb2z" (UID: "bda12f0d-5325-4258-a601-381a0751907a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:19:30.564485 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-etc-kubernetes\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.564485 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563682    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-node-log\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.565165 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563712    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.565165 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563718    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-etc-modprobe-d\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.565165 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563722    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/138735f1-3e23-48e4-88a5-7992d2987d07-registration-dir\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.565165 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563743    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45d555da-de43-4a10-85d3-b24a77d99118-ovnkube-script-lib\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.565165 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563761    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-host\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.565165 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-etc-systemd\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.565165 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-log-socket\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.565165 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563835    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1dc4198c-786b-4dec-abb0-0838e3fab08a-agent-certs\") pod \"konnectivity-agent-hrsjh\" (UID: \"1dc4198c-786b-4dec-abb0-0838e3fab08a\") " pod="kube-system/konnectivity-agent-hrsjh"
Jul 24 12:19:30.565165 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-multus-cni-dir\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.565165 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-etc-sysconfig\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.565165 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0e54908-d259-4a41-b758-eb167ca6239c-tmp\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.565165 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563942    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-kubelet\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.565165 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563967    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45d555da-de43-4a10-85d3-b24a77d99118-ovnkube-config\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.565165 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563994    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1659a0ea-58c5-41f5-b8a6-b59ac3ae0ef3-host-slash\") pod \"iptables-alerter-k6gfj\" (UID: \"1659a0ea-58c5-41f5-b8a6-b59ac3ae0ef3\") " pod="openshift-network-operator/iptables-alerter-k6gfj"
Jul 24 12:19:30.565165 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.564009    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-host-var-lib-kubelet\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.565165 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.564023    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret\") pod \"global-pull-secret-syncer-qrzvj\" (UID: \"d98f89d0-11ef-4c6b-b973-64267283f1da\") " pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:30.565165 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.564038    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0385bad7-43a0-4ee3-b116-0e89d51f30f7-tmp-dir\") pod \"node-resolver-jmmcs\" (UID: \"0385bad7-43a0-4ee3-b116-0e89d51f30f7\") " pod="openshift-dns/node-resolver-jmmcs"
Jul 24 12:19:30.566086 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.564048    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-multus-conf-dir\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.566086 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.564089    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/138735f1-3e23-48e4-88a5-7992d2987d07-device-dir\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.566086 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.564101    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-host-var-lib-cni-multus\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.566086 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.564364    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-etc-systemd\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.566086 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.564497    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-etc-sysctl-d\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.566086 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.564513    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-etc-sysconfig\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.566086 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.564608    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-host-run-multus-certs\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.566086 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563808    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-os-release\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.566086 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.564845    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a0e54908-d259-4a41-b758-eb167ca6239c-run\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.566086 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.563966    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 12:19:30.566086 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.564876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-multus-cni-dir\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.566086 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.564967    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a576aeb-3eff-4419-9bee-f606af167d49-cni-binary-copy\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.566086 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.565198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a576aeb-3eff-4419-9bee-f606af167d49-host-var-lib-cni-bin\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.569522 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.569500    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a0e54908-d259-4a41-b758-eb167ca6239c-etc-tuned\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.569705 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.569638    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0e54908-d259-4a41-b758-eb167ca6239c-tmp\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.569838 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.569817    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1dc4198c-786b-4dec-abb0-0838e3fab08a-agent-certs\") pod \"konnectivity-agent-hrsjh\" (UID: \"1dc4198c-786b-4dec-abb0-0838e3fab08a\") " pod="kube-system/konnectivity-agent-hrsjh"
Jul 24 12:19:30.574332 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.574313    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g9tk\" (UniqueName: \"kubernetes.io/projected/3a576aeb-3eff-4419-9bee-f606af167d49-kube-api-access-4g9tk\") pod \"multus-jzmtg\" (UID: \"3a576aeb-3eff-4419-9bee-f606af167d49\") " pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.579110 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:30.579018    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:19:30.579110 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:30.579072    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:19:30.579110 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:30.579086    2581 projected.go:196] Error preparing data for projected volume kube-api-access-bm2k2 for pod openshift-network-diagnostics/network-check-target-8294s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:19:30.579110 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:30.579209    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2 podName:edc2499b-10da-416b-831d-52f00cbd026e nodeName:}" failed. No retries permitted until 2026-07-24 12:19:31.079181571 +0000 UTC m=+3.034186315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bm2k2" (UniqueName: "kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2") pod "network-check-target-8294s" (UID: "edc2499b-10da-416b-831d-52f00cbd026e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:19:30.581991 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.581944    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r7dz\" (UniqueName: \"kubernetes.io/projected/0385bad7-43a0-4ee3-b116-0e89d51f30f7-kube-api-access-9r7dz\") pod \"node-resolver-jmmcs\" (UID: \"0385bad7-43a0-4ee3-b116-0e89d51f30f7\") " pod="openshift-dns/node-resolver-jmmcs"
Jul 24 12:19:30.582140 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.582123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lvqp\" (UniqueName: \"kubernetes.io/projected/bda12f0d-5325-4258-a601-381a0751907a-kube-api-access-8lvqp\") pod \"network-metrics-daemon-wtb2z\" (UID: \"bda12f0d-5325-4258-a601-381a0751907a\") " pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:30.583240 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.583219    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qjk9\" (UniqueName: \"kubernetes.io/projected/138735f1-3e23-48e4-88a5-7992d2987d07-kube-api-access-7qjk9\") pod \"aws-ebs-csi-driver-node-sp62z\" (UID: \"138735f1-3e23-48e4-88a5-7992d2987d07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.585019 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.585002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq2r9\" (UniqueName: \"kubernetes.io/projected/a0e54908-d259-4a41-b758-eb167ca6239c-kube-api-access-zq2r9\") pod \"tuned-tvdgf\" (UID: \"a0e54908-d259-4a41-b758-eb167ca6239c\") " pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.665047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.664999    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.665047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c74cea6b-83f7-4615-918a-d27808ed55ee-serviceca\") pod \"node-ca-sj8p9\" (UID: \"c74cea6b-83f7-4615-918a-d27808ed55ee\") " pod="openshift-image-registry/node-ca-sj8p9"
Jul 24 12:19:30.665247 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-etc-openvswitch\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.665247 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665124    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-etc-openvswitch\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.665341 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-run-ovn\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.665341 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665287    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45d555da-de43-4a10-85d3-b24a77d99118-ovn-node-metrics-cert\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.665341 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665333    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1659a0ea-58c5-41f5-b8a6-b59ac3ae0ef3-iptables-alerter-script\") pod \"iptables-alerter-k6gfj\" (UID: \"1659a0ea-58c5-41f5-b8a6-b59ac3ae0ef3\") " pod="openshift-network-operator/iptables-alerter-k6gfj"
Jul 24 12:19:30.665500 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv64b\" (UniqueName: \"kubernetes.io/projected/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-kube-api-access-mv64b\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.665500 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-systemd-units\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.665500 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-node-log\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.665500 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665451    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.665500 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45d555da-de43-4a10-85d3-b24a77d99118-ovnkube-script-lib\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.665500 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-log-socket\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.665773 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-kubelet\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.665773 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45d555da-de43-4a10-85d3-b24a77d99118-ovnkube-config\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.665773 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665568    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1659a0ea-58c5-41f5-b8a6-b59ac3ae0ef3-host-slash\") pod \"iptables-alerter-k6gfj\" (UID: \"1659a0ea-58c5-41f5-b8a6-b59ac3ae0ef3\") " pod="openshift-network-operator/iptables-alerter-k6gfj"
Jul 24 12:19:30.665773 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665574    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c74cea6b-83f7-4615-918a-d27808ed55ee-serviceca\") pod \"node-ca-sj8p9\" (UID: \"c74cea6b-83f7-4615-918a-d27808ed55ee\") " pod="openshift-image-registry/node-ca-sj8p9"
Jul 24 12:19:30.665773 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665590    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret\") pod \"global-pull-secret-syncer-qrzvj\" (UID: \"d98f89d0-11ef-4c6b-b973-64267283f1da\") " pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:30.665773 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.665773 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665654    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.665773 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-cni-bin\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.665773 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:30.665671    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:19:30.665773 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4zb8\" (UniqueName: \"kubernetes.io/projected/45d555da-de43-4a10-85d3-b24a77d99118-kube-api-access-l4zb8\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.665773 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.665773 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:30.665735    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret podName:d98f89d0-11ef-4c6b-b973-64267283f1da nodeName:}" failed. No retries permitted until 2026-07-24 12:19:31.165718172 +0000 UTC m=+3.120722983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret") pod "global-pull-secret-syncer-qrzvj" (UID: "d98f89d0-11ef-4c6b-b973-64267283f1da") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:19:30.665773 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.665335    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-run-ovn\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.666228 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666041    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.666228 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-cni-bin\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.666295 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666235    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1659a0ea-58c5-41f5-b8a6-b59ac3ae0ef3-iptables-alerter-script\") pod \"iptables-alerter-k6gfj\" (UID: \"1659a0ea-58c5-41f5-b8a6-b59ac3ae0ef3\") " pod="openshift-network-operator/iptables-alerter-k6gfj"
Jul 24 12:19:30.666295 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666245    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-systemd-units\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.666295 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666279    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-node-log\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.666821 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666304    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-log-socket\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.666821 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666320    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-kubelet\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.666821 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666342    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1659a0ea-58c5-41f5-b8a6-b59ac3ae0ef3-host-slash\") pod \"iptables-alerter-k6gfj\" (UID: \"1659a0ea-58c5-41f5-b8a6-b59ac3ae0ef3\") " pod="openshift-network-operator/iptables-alerter-k6gfj"
Jul 24 12:19:30.666821 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x25gj\" (UniqueName: \"kubernetes.io/projected/c74cea6b-83f7-4615-918a-d27808ed55ee-kube-api-access-x25gj\") pod \"node-ca-sj8p9\" (UID: \"c74cea6b-83f7-4615-918a-d27808ed55ee\") " pod="openshift-image-registry/node-ca-sj8p9"
Jul 24 12:19:30.666821 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.666821 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666432    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d98f89d0-11ef-4c6b-b973-64267283f1da-dbus\") pod \"global-pull-secret-syncer-qrzvj\" (UID: \"d98f89d0-11ef-4c6b-b973-64267283f1da\") " pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:30.666821 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgrnl\" (UniqueName: \"kubernetes.io/projected/1659a0ea-58c5-41f5-b8a6-b59ac3ae0ef3-kube-api-access-zgrnl\") pod \"iptables-alerter-k6gfj\" (UID: \"1659a0ea-58c5-41f5-b8a6-b59ac3ae0ef3\") " pod="openshift-network-operator/iptables-alerter-k6gfj"
Jul 24 12:19:30.666821 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-cnibin\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.666821 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666502    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d98f89d0-11ef-4c6b-b973-64267283f1da-kubelet-config\") pod \"global-pull-secret-syncer-qrzvj\" (UID: \"d98f89d0-11ef-4c6b-b973-64267283f1da\") " pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:30.666821 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666523    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-slash\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.666821 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-run-ovn-kubernetes\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.666821 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-var-lib-openvswitch\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.666821 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666601    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d98f89d0-11ef-4c6b-b973-64267283f1da-kubelet-config\") pod \"global-pull-secret-syncer-qrzvj\" (UID: \"d98f89d0-11ef-4c6b-b973-64267283f1da\") " pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:30.666821 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c74cea6b-83f7-4615-918a-d27808ed55ee-host\") pod \"node-ca-sj8p9\" (UID: \"c74cea6b-83f7-4615-918a-d27808ed55ee\") " pod="openshift-image-registry/node-ca-sj8p9"
Jul 24 12:19:30.666821 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666646    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d98f89d0-11ef-4c6b-b973-64267283f1da-dbus\") pod \"global-pull-secret-syncer-qrzvj\" (UID: \"d98f89d0-11ef-4c6b-b973-64267283f1da\") " pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:30.666821 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-cnibin\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.666821 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-os-release\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.667410 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c74cea6b-83f7-4615-918a-d27808ed55ee-host\") pod \"node-ca-sj8p9\" (UID: \"c74cea6b-83f7-4615-918a-d27808ed55ee\") " pod="openshift-image-registry/node-ca-sj8p9"
Jul 24 12:19:30.667410 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666728    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-run-netns\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.667410 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666734    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-slash\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.667410 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666738    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-var-lib-openvswitch\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.667410 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666754    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45d555da-de43-4a10-85d3-b24a77d99118-ovnkube-config\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.667410 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666769    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-run-netns\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.667410 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666697    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-run-ovn-kubernetes\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.667410 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-run-systemd\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.667410 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45d555da-de43-4a10-85d3-b24a77d99118-ovnkube-script-lib\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.667410 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666795    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-os-release\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.667410 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-run-openvswitch\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.667410 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666834    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-cni-netd\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.667410 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666839    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-run-systemd\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.667410 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.667410 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666889    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-run-openvswitch\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.667410 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666922    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-system-cni-dir\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.667410 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45d555da-de43-4a10-85d3-b24a77d99118-host-cni-netd\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.668036 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666927    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.668036 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666962    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-system-cni-dir\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.668036 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.666987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45d555da-de43-4a10-85d3-b24a77d99118-env-overrides\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.668036 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.667342    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45d555da-de43-4a10-85d3-b24a77d99118-env-overrides\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.668036 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.667357    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.669585 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.669564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45d555da-de43-4a10-85d3-b24a77d99118-ovn-node-metrics-cert\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.680964 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.680910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv64b\" (UniqueName: \"kubernetes.io/projected/f42ba11d-0506-4915-bc77-d52d5fd3ea4a-kube-api-access-mv64b\") pod \"multus-additional-cni-plugins-bk4w6\" (UID: \"f42ba11d-0506-4915-bc77-d52d5fd3ea4a\") " pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.682235 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.682206    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4zb8\" (UniqueName: \"kubernetes.io/projected/45d555da-de43-4a10-85d3-b24a77d99118-kube-api-access-l4zb8\") pod \"ovnkube-node-tzz5k\" (UID: \"45d555da-de43-4a10-85d3-b24a77d99118\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.685150 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.685116    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgrnl\" (UniqueName: \"kubernetes.io/projected/1659a0ea-58c5-41f5-b8a6-b59ac3ae0ef3-kube-api-access-zgrnl\") pod \"iptables-alerter-k6gfj\" (UID: \"1659a0ea-58c5-41f5-b8a6-b59ac3ae0ef3\") " pod="openshift-network-operator/iptables-alerter-k6gfj"
Jul 24 12:19:30.687981 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.687964    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x25gj\" (UniqueName: \"kubernetes.io/projected/c74cea6b-83f7-4615-918a-d27808ed55ee-kube-api-access-x25gj\") pod \"node-ca-sj8p9\" (UID: \"c74cea6b-83f7-4615-918a-d27808ed55ee\") " pod="openshift-image-registry/node-ca-sj8p9"
Jul 24 12:19:30.753189 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.753151    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tvdgf"
Jul 24 12:19:30.768226 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.768197    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hrsjh"
Jul 24 12:19:30.779247 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.779221    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z"
Jul 24 12:19:30.784046 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.784028    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jmmcs"
Jul 24 12:19:30.793926 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.793900    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jzmtg"
Jul 24 12:19:30.794529 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.794499    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 12:19:30.802259 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.802238    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bk4w6"
Jul 24 12:19:30.817054 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.817030    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:30.823828 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.823809    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sj8p9"
Jul 24 12:19:30.830439 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:30.830399    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k6gfj"
Jul 24 12:19:31.070708 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:31.070672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs\") pod \"network-metrics-daemon-wtb2z\" (UID: \"bda12f0d-5325-4258-a601-381a0751907a\") " pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:31.070894 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:31.070799    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:19:31.070894 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:31.070860    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs podName:bda12f0d-5325-4258-a601-381a0751907a nodeName:}" failed. No retries permitted until 2026-07-24 12:19:32.07084374 +0000 UTC m=+4.025848487 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs") pod "network-metrics-daemon-wtb2z" (UID: "bda12f0d-5325-4258-a601-381a0751907a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:19:31.171115 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:31.171082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret\") pod \"global-pull-secret-syncer-qrzvj\" (UID: \"d98f89d0-11ef-4c6b-b973-64267283f1da\") " pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:31.171296 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:31.171158    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm2k2\" (UniqueName: \"kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2\") pod \"network-check-target-8294s\" (UID: \"edc2499b-10da-416b-831d-52f00cbd026e\") " pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:31.171296 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:31.171180    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:19:31.171296 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:31.171232    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret podName:d98f89d0-11ef-4c6b-b973-64267283f1da nodeName:}" failed. No retries permitted until 2026-07-24 12:19:32.171217613 +0000 UTC m=+4.126222354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret") pod "global-pull-secret-syncer-qrzvj" (UID: "d98f89d0-11ef-4c6b-b973-64267283f1da") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:19:31.171296 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:31.171275    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:19:31.171296 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:31.171290    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:19:31.171296 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:31.171302    2581 projected.go:196] Error preparing data for projected volume kube-api-access-bm2k2 for pod openshift-network-diagnostics/network-check-target-8294s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:19:31.171622 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:31.171356    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2 podName:edc2499b-10da-416b-831d-52f00cbd026e nodeName:}" failed. No retries permitted until 2026-07-24 12:19:32.171341143 +0000 UTC m=+4.126345884 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bm2k2" (UniqueName: "kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2") pod "network-check-target-8294s" (UID: "edc2499b-10da-416b-831d-52f00cbd026e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:19:31.508085 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:31.508047    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 12:14:30 +0000 UTC" deadline="2028-03-29 02:36:48.948403123 +0000 UTC"
Jul 24 12:19:31.508085 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:31.508081    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14726h17m17.44032561s"
Jul 24 12:19:31.557278 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:31.557097    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0385bad7_43a0_4ee3_b116_0e89d51f30f7.slice/crio-aee2829a399b97ff5059756c1ba57c7f56d36fd55b53dac58f22234ed87f9e82 WatchSource:0}: Error finding container aee2829a399b97ff5059756c1ba57c7f56d36fd55b53dac58f22234ed87f9e82: Status 404 returned error can't find the container with id aee2829a399b97ff5059756c1ba57c7f56d36fd55b53dac58f22234ed87f9e82
Jul 24 12:19:31.558632 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:31.558605    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e54908_d259_4a41_b758_eb167ca6239c.slice/crio-e72336f114793c5364b2c97ac759395a4941c1e320396ed7227964138155ddf5 WatchSource:0}: Error finding container e72336f114793c5364b2c97ac759395a4941c1e320396ed7227964138155ddf5: Status 404 returned error can't find the container with id e72336f114793c5364b2c97ac759395a4941c1e320396ed7227964138155ddf5
Jul 24 12:19:31.559932 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:31.559908    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1659a0ea_58c5_41f5_b8a6_b59ac3ae0ef3.slice/crio-4f1a72d1d3dec379a6f4e86e51894e96a0bd51ecb485a971be03568190087db4 WatchSource:0}: Error finding container 4f1a72d1d3dec379a6f4e86e51894e96a0bd51ecb485a971be03568190087db4: Status 404 returned error can't find the container with id 4f1a72d1d3dec379a6f4e86e51894e96a0bd51ecb485a971be03568190087db4
Jul 24 12:19:31.561956 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:31.561931    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74cea6b_83f7_4615_918a_d27808ed55ee.slice/crio-11416eecd30eede760269c32f910ec958166591542a97f483cb3f39caff954b0 WatchSource:0}: Error finding container 11416eecd30eede760269c32f910ec958166591542a97f483cb3f39caff954b0: Status 404 returned error can't find the container with id 11416eecd30eede760269c32f910ec958166591542a97f483cb3f39caff954b0
Jul 24 12:19:31.566833 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:31.566810    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a576aeb_3eff_4419_9bee_f606af167d49.slice/crio-9b147031c519291b6bb74789d1eb6c9224c622c9683d52a4a2a73e0911ff73b8 WatchSource:0}: Error finding container 9b147031c519291b6bb74789d1eb6c9224c622c9683d52a4a2a73e0911ff73b8: Status 404 returned error can't find the container with id 9b147031c519291b6bb74789d1eb6c9224c622c9683d52a4a2a73e0911ff73b8
Jul 24 12:19:31.568596 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:31.568530    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc4198c_786b_4dec_abb0_0838e3fab08a.slice/crio-ad6d374ec6ebec1cccd9f9d167ab6c1f03811584398294ffb666b32bcd85f24e WatchSource:0}: Error finding container ad6d374ec6ebec1cccd9f9d167ab6c1f03811584398294ffb666b32bcd85f24e: Status 404 returned error can't find the container with id ad6d374ec6ebec1cccd9f9d167ab6c1f03811584398294ffb666b32bcd85f24e
Jul 24 12:19:31.570805 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:19:31.570692    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d555da_de43_4a10_85d3_b24a77d99118.slice/crio-0cfd8e096f780464afe527d36a1af48a828d6a155d7db032983d06b04c55faa7 WatchSource:0}: Error finding container 0cfd8e096f780464afe527d36a1af48a828d6a155d7db032983d06b04c55faa7: Status 404 returned error can't find the container with id 0cfd8e096f780464afe527d36a1af48a828d6a155d7db032983d06b04c55faa7
Jul 24 12:19:31.650550 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:31.650338    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-194.ec2.internal" event={"ID":"69ea8b15ea96888b6c85fb2084a0817d","Type":"ContainerStarted","Data":"b59c7c8c2900d13340eefaa34c1f95bb7c3351a40c962a2973826e1b742824a2"}
Jul 24 12:19:31.651402 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:31.651380    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hrsjh" event={"ID":"1dc4198c-786b-4dec-abb0-0838e3fab08a","Type":"ContainerStarted","Data":"ad6d374ec6ebec1cccd9f9d167ab6c1f03811584398294ffb666b32bcd85f24e"}
Jul 24 12:19:31.655720 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:31.655688    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z" event={"ID":"138735f1-3e23-48e4-88a5-7992d2987d07","Type":"ContainerStarted","Data":"5ae8992a31102a9a78f7c11ad374e890afdc1af4f5d31052b294dba7ca97b8d8"}
Jul 24 12:19:31.659946 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:31.659926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k" event={"ID":"45d555da-de43-4a10-85d3-b24a77d99118","Type":"ContainerStarted","Data":"0cfd8e096f780464afe527d36a1af48a828d6a155d7db032983d06b04c55faa7"}
Jul 24 12:19:31.662739 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:31.662718    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jzmtg" event={"ID":"3a576aeb-3eff-4419-9bee-f606af167d49","Type":"ContainerStarted","Data":"9b147031c519291b6bb74789d1eb6c9224c622c9683d52a4a2a73e0911ff73b8"}
Jul 24 12:19:31.665929 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:31.665909    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bk4w6" event={"ID":"f42ba11d-0506-4915-bc77-d52d5fd3ea4a","Type":"ContainerStarted","Data":"0567e4f900a83c03194208974723781ec8b28594651dc8a9eb2ced0e306fa796"}
Jul 24 12:19:31.667005 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:31.666976    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sj8p9" event={"ID":"c74cea6b-83f7-4615-918a-d27808ed55ee","Type":"ContainerStarted","Data":"11416eecd30eede760269c32f910ec958166591542a97f483cb3f39caff954b0"}
Jul 24 12:19:31.668429 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:31.668398    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k6gfj" event={"ID":"1659a0ea-58c5-41f5-b8a6-b59ac3ae0ef3","Type":"ContainerStarted","Data":"4f1a72d1d3dec379a6f4e86e51894e96a0bd51ecb485a971be03568190087db4"}
Jul 24 12:19:31.669454 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:31.669436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tvdgf" event={"ID":"a0e54908-d259-4a41-b758-eb167ca6239c","Type":"ContainerStarted","Data":"e72336f114793c5364b2c97ac759395a4941c1e320396ed7227964138155ddf5"}
Jul 24 12:19:31.671222 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:31.671148    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jmmcs" event={"ID":"0385bad7-43a0-4ee3-b116-0e89d51f30f7","Type":"ContainerStarted","Data":"aee2829a399b97ff5059756c1ba57c7f56d36fd55b53dac58f22234ed87f9e82"}
Jul 24 12:19:32.079753 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:32.078404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs\") pod \"network-metrics-daemon-wtb2z\" (UID: \"bda12f0d-5325-4258-a601-381a0751907a\") " pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:32.079753 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:32.078611    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:19:32.079753 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:32.078690    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs podName:bda12f0d-5325-4258-a601-381a0751907a nodeName:}" failed. No retries permitted until 2026-07-24 12:19:34.078669343 +0000 UTC m=+6.033674100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs") pod "network-metrics-daemon-wtb2z" (UID: "bda12f0d-5325-4258-a601-381a0751907a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:19:32.179057 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:32.179012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret\") pod \"global-pull-secret-syncer-qrzvj\" (UID: \"d98f89d0-11ef-4c6b-b973-64267283f1da\") " pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:32.179223 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:32.179095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm2k2\" (UniqueName: \"kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2\") pod \"network-check-target-8294s\" (UID: \"edc2499b-10da-416b-831d-52f00cbd026e\") " pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:32.179297 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:32.179245    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:19:32.179297 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:32.179264    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:19:32.179297 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:32.179276    2581 projected.go:196] Error preparing data for projected volume kube-api-access-bm2k2 for pod openshift-network-diagnostics/network-check-target-8294s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:19:32.179452 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:32.179334    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2 podName:edc2499b-10da-416b-831d-52f00cbd026e nodeName:}" failed. No retries permitted until 2026-07-24 12:19:34.1793155 +0000 UTC m=+6.134320261 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bm2k2" (UniqueName: "kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2") pod "network-check-target-8294s" (UID: "edc2499b-10da-416b-831d-52f00cbd026e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:19:32.179779 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:32.179758    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:19:32.179869 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:32.179816    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret podName:d98f89d0-11ef-4c6b-b973-64267283f1da nodeName:}" failed. No retries permitted until 2026-07-24 12:19:34.17980006 +0000 UTC m=+6.134804816 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret") pod "global-pull-secret-syncer-qrzvj" (UID: "d98f89d0-11ef-4c6b-b973-64267283f1da") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:19:32.508325 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:32.508223    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 12:14:30 +0000 UTC" deadline="2028-02-14 08:55:08.96047082 +0000 UTC"
Jul 24 12:19:32.508325 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:32.508282    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13676h35m36.452192351s"
Jul 24 12:19:32.643319 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:32.642475    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:32.643319 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:32.642615    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtb2z" podUID="bda12f0d-5325-4258-a601-381a0751907a"
Jul 24 12:19:32.643319 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:32.643020    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:32.643319 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:32.643140    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8294s" podUID="edc2499b-10da-416b-831d-52f00cbd026e"
Jul 24 12:19:32.643319 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:32.643198    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:32.643319 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:32.643272    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrzvj" podUID="d98f89d0-11ef-4c6b-b973-64267283f1da"
Jul 24 12:19:32.695312 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:32.695099    2581 generic.go:356] "Generic (PLEG): container finished" podID="3643e41074a2c5417adb5993a4721adf" containerID="00f8900801016dd554ca5b27945a3f7c217dca81e27826b03dbfaea9882a6c44" exitCode=0
Jul 24 12:19:32.696145 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:32.696111    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal" event={"ID":"3643e41074a2c5417adb5993a4721adf","Type":"ContainerDied","Data":"00f8900801016dd554ca5b27945a3f7c217dca81e27826b03dbfaea9882a6c44"}
Jul 24 12:19:32.714206 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:32.714153    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-194.ec2.internal" podStartSLOduration=3.714133134 podStartE2EDuration="3.714133134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:31.66767141 +0000 UTC m=+3.622676173" watchObservedRunningTime="2026-07-24 12:19:32.714133134 +0000 UTC m=+4.669137899"
Jul 24 12:19:33.718579 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:33.718506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal" event={"ID":"3643e41074a2c5417adb5993a4721adf","Type":"ContainerStarted","Data":"5c5a822f89f84d8b64d4c096884c494e04d0eef89d245a94a6b8e8256b193eea"}
Jul 24 12:19:34.098525 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:34.098473    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs\") pod \"network-metrics-daemon-wtb2z\" (UID: \"bda12f0d-5325-4258-a601-381a0751907a\") " pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:34.098702 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:34.098679    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:19:34.098773 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:34.098742    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs podName:bda12f0d-5325-4258-a601-381a0751907a nodeName:}" failed. No retries permitted until 2026-07-24 12:19:38.098724182 +0000 UTC m=+10.053728927 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs") pod "network-metrics-daemon-wtb2z" (UID: "bda12f0d-5325-4258-a601-381a0751907a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:19:34.199602 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:34.199563    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm2k2\" (UniqueName: \"kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2\") pod \"network-check-target-8294s\" (UID: \"edc2499b-10da-416b-831d-52f00cbd026e\") " pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:34.199769 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:34.199648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret\") pod \"global-pull-secret-syncer-qrzvj\" (UID: \"d98f89d0-11ef-4c6b-b973-64267283f1da\") " pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:34.199853 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:34.199827    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:19:34.199906 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:34.199894    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret podName:d98f89d0-11ef-4c6b-b973-64267283f1da nodeName:}" failed. No retries permitted until 2026-07-24 12:19:38.199873796 +0000 UTC m=+10.154878540 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret") pod "global-pull-secret-syncer-qrzvj" (UID: "d98f89d0-11ef-4c6b-b973-64267283f1da") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:19:34.200036 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:34.200021    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:19:34.200100 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:34.200042    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:19:34.200100 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:34.200055    2581 projected.go:196] Error preparing data for projected volume kube-api-access-bm2k2 for pod openshift-network-diagnostics/network-check-target-8294s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:19:34.200100 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:34.200095    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2 podName:edc2499b-10da-416b-831d-52f00cbd026e nodeName:}" failed. No retries permitted until 2026-07-24 12:19:38.200082996 +0000 UTC m=+10.155087751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bm2k2" (UniqueName: "kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2") pod "network-check-target-8294s" (UID: "edc2499b-10da-416b-831d-52f00cbd026e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:19:34.639664 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:34.639632    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:34.639846 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:34.639774    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtb2z" podUID="bda12f0d-5325-4258-a601-381a0751907a"
Jul 24 12:19:34.640433 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:34.640399    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:34.640541 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:34.640505    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8294s" podUID="edc2499b-10da-416b-831d-52f00cbd026e"
Jul 24 12:19:34.641878 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:34.641855    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:34.642002 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:34.641957    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrzvj" podUID="d98f89d0-11ef-4c6b-b973-64267283f1da"
Jul 24 12:19:36.642829 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:36.639646    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:36.642829 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:36.639782    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8294s" podUID="edc2499b-10da-416b-831d-52f00cbd026e"
Jul 24 12:19:36.642829 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:36.642521    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:36.642829 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:36.642641    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtb2z" podUID="bda12f0d-5325-4258-a601-381a0751907a"
Jul 24 12:19:36.642829 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:36.639602    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:36.642829 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:36.642751    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrzvj" podUID="d98f89d0-11ef-4c6b-b973-64267283f1da"
Jul 24 12:19:38.139575 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:38.139454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs\") pod \"network-metrics-daemon-wtb2z\" (UID: \"bda12f0d-5325-4258-a601-381a0751907a\") " pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:38.140005 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:38.139648    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:19:38.140005 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:38.139723    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs podName:bda12f0d-5325-4258-a601-381a0751907a nodeName:}" failed. No retries permitted until 2026-07-24 12:19:46.139707815 +0000 UTC m=+18.094712556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs") pod "network-metrics-daemon-wtb2z" (UID: "bda12f0d-5325-4258-a601-381a0751907a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:19:38.240406 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:38.240355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret\") pod \"global-pull-secret-syncer-qrzvj\" (UID: \"d98f89d0-11ef-4c6b-b973-64267283f1da\") " pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:38.240608 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:38.240512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm2k2\" (UniqueName: \"kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2\") pod \"network-check-target-8294s\" (UID: \"edc2499b-10da-416b-831d-52f00cbd026e\") " pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:38.240766 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:38.240744    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:19:38.240860 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:38.240763    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:19:38.240860 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:38.240843    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret podName:d98f89d0-11ef-4c6b-b973-64267283f1da nodeName:}" failed. No retries permitted until 2026-07-24 12:19:46.240822653 +0000 UTC m=+18.195827399 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret") pod "global-pull-secret-syncer-qrzvj" (UID: "d98f89d0-11ef-4c6b-b973-64267283f1da") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:19:38.240970 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:38.240774    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:19:38.240970 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:38.240887    2581 projected.go:196] Error preparing data for projected volume kube-api-access-bm2k2 for pod openshift-network-diagnostics/network-check-target-8294s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:19:38.240970 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:38.240941    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2 podName:edc2499b-10da-416b-831d-52f00cbd026e nodeName:}" failed. No retries permitted until 2026-07-24 12:19:46.240923548 +0000 UTC m=+18.195928309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bm2k2" (UniqueName: "kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2") pod "network-check-target-8294s" (UID: "edc2499b-10da-416b-831d-52f00cbd026e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:19:38.640720 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:38.640690    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:38.640880 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:38.640775    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrzvj" podUID="d98f89d0-11ef-4c6b-b973-64267283f1da"
Jul 24 12:19:38.640880 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:38.640848    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:38.640953 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:38.640931    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8294s" podUID="edc2499b-10da-416b-831d-52f00cbd026e"
Jul 24 12:19:38.640953 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:38.640938    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:38.641080 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:38.641047    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtb2z" podUID="bda12f0d-5325-4258-a601-381a0751907a"
Jul 24 12:19:40.639447 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:40.639395    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:40.639905 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:40.639395    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:40.639905 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:40.639564    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtb2z" podUID="bda12f0d-5325-4258-a601-381a0751907a"
Jul 24 12:19:40.639905 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:40.639655    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8294s" podUID="edc2499b-10da-416b-831d-52f00cbd026e"
Jul 24 12:19:40.639905 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:40.639401    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:40.639905 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:40.639755    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrzvj" podUID="d98f89d0-11ef-4c6b-b973-64267283f1da"
Jul 24 12:19:42.640397 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:42.640353    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:42.640888 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:42.640353    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:42.640888 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:42.640362    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:42.640888 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:42.640533    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtb2z" podUID="bda12f0d-5325-4258-a601-381a0751907a"
Jul 24 12:19:42.640888 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:42.640662    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrzvj" podUID="d98f89d0-11ef-4c6b-b973-64267283f1da"
Jul 24 12:19:42.640888 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:42.640743    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8294s" podUID="edc2499b-10da-416b-831d-52f00cbd026e"
Jul 24 12:19:44.639865 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:44.639825    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:44.639865 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:44.639854    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:44.640405 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:44.639870    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:44.640405 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:44.639980    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtb2z" podUID="bda12f0d-5325-4258-a601-381a0751907a"
Jul 24 12:19:44.640405 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:44.640084    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrzvj" podUID="d98f89d0-11ef-4c6b-b973-64267283f1da"
Jul 24 12:19:44.640405 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:44.640166    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8294s" podUID="edc2499b-10da-416b-831d-52f00cbd026e"
Jul 24 12:19:46.203668 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:46.203635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs\") pod \"network-metrics-daemon-wtb2z\" (UID: \"bda12f0d-5325-4258-a601-381a0751907a\") " pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:46.204099 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:46.203786    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:19:46.204099 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:46.203853    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs podName:bda12f0d-5325-4258-a601-381a0751907a nodeName:}" failed. No retries permitted until 2026-07-24 12:20:02.203835201 +0000 UTC m=+34.158839944 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs") pod "network-metrics-daemon-wtb2z" (UID: "bda12f0d-5325-4258-a601-381a0751907a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:19:46.304191 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:46.304153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm2k2\" (UniqueName: \"kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2\") pod \"network-check-target-8294s\" (UID: \"edc2499b-10da-416b-831d-52f00cbd026e\") " pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:46.304363 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:46.304223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret\") pod \"global-pull-secret-syncer-qrzvj\" (UID: \"d98f89d0-11ef-4c6b-b973-64267283f1da\") " pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:46.304363 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:46.304334    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:19:46.304363 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:46.304348    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:19:46.304571 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:46.304372    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:19:46.304571 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:46.304386    2581 projected.go:196] Error preparing data for projected volume kube-api-access-bm2k2 for pod openshift-network-diagnostics/network-check-target-8294s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:19:46.304571 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:46.304429    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret podName:d98f89d0-11ef-4c6b-b973-64267283f1da nodeName:}" failed. No retries permitted until 2026-07-24 12:20:02.304392749 +0000 UTC m=+34.259397517 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret") pod "global-pull-secret-syncer-qrzvj" (UID: "d98f89d0-11ef-4c6b-b973-64267283f1da") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:19:46.304571 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:46.304454    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2 podName:edc2499b-10da-416b-831d-52f00cbd026e nodeName:}" failed. No retries permitted until 2026-07-24 12:20:02.304439195 +0000 UTC m=+34.259443952 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bm2k2" (UniqueName: "kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2") pod "network-check-target-8294s" (UID: "edc2499b-10da-416b-831d-52f00cbd026e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:19:46.639974 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:46.639935    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:46.639974 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:46.639973    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:46.640227 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:46.639935    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:46.640227 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:46.640072    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8294s" podUID="edc2499b-10da-416b-831d-52f00cbd026e"
Jul 24 12:19:46.640227 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:46.640183    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtb2z" podUID="bda12f0d-5325-4258-a601-381a0751907a"
Jul 24 12:19:46.640366 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:46.640266    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrzvj" podUID="d98f89d0-11ef-4c6b-b973-64267283f1da"
Jul 24 12:19:48.640901 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:48.640397    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:48.640901 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:48.640511    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtb2z" podUID="bda12f0d-5325-4258-a601-381a0751907a"
Jul 24 12:19:48.640901 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:48.640524    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:48.640901 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:48.640644    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrzvj" podUID="d98f89d0-11ef-4c6b-b973-64267283f1da"
Jul 24 12:19:48.640901 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:48.640680    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:48.640901 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:48.640740    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8294s" podUID="edc2499b-10da-416b-831d-52f00cbd026e"
Jul 24 12:19:50.640812 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:50.640548    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:50.641406 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:50.640935    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8294s" podUID="edc2499b-10da-416b-831d-52f00cbd026e"
Jul 24 12:19:50.641406 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:50.641018    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:50.641406 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:50.641031    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:50.641406 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:50.641134    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrzvj" podUID="d98f89d0-11ef-4c6b-b973-64267283f1da"
Jul 24 12:19:50.641406 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:50.641232    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtb2z" podUID="bda12f0d-5325-4258-a601-381a0751907a"
Jul 24 12:19:50.751966 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:50.751725    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tvdgf" event={"ID":"a0e54908-d259-4a41-b758-eb167ca6239c","Type":"ContainerStarted","Data":"ff0e30a790d11455504b46a4039f2ce162b22fc1453ac0a296a5e4554fb76df3"}
Jul 24 12:19:50.753277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:50.753244    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jmmcs" event={"ID":"0385bad7-43a0-4ee3-b116-0e89d51f30f7","Type":"ContainerStarted","Data":"2850a417fc4b782aaf98db43f4cf8edb8064117c9a60c407aa21ecd7b924c337"}
Jul 24 12:19:50.754613 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:50.754587    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hrsjh" event={"ID":"1dc4198c-786b-4dec-abb0-0838e3fab08a","Type":"ContainerStarted","Data":"44e2efe7c7c21bddcbb22f13ddecdcee1dccc69ca7c3ac8692a4e13112339daa"}
Jul 24 12:19:50.756216 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:50.756191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z" event={"ID":"138735f1-3e23-48e4-88a5-7992d2987d07","Type":"ContainerStarted","Data":"6b5385183ee6ebdfc84f8280f30e02064f386ce3882e4b54cb90b5f2bc05258e"}
Jul 24 12:19:50.757959 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:50.757937    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k" event={"ID":"45d555da-de43-4a10-85d3-b24a77d99118","Type":"ContainerStarted","Data":"c78ed1c3bf17519dcbf78556a0745ec88c46b4fd6346996420cd8757930f345a"}
Jul 24 12:19:50.759776 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:50.759625    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jzmtg" event={"ID":"3a576aeb-3eff-4419-9bee-f606af167d49","Type":"ContainerStarted","Data":"721142b6750b51a571d9517e701dd1cb1f220dd53cf3ee1432943643ed5da021"}
Jul 24 12:19:50.761069 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:50.761043    2581 generic.go:356] "Generic (PLEG): container finished" podID="f42ba11d-0506-4915-bc77-d52d5fd3ea4a" containerID="1a72984584af6f0bb24923ea549f6ead47fe5c50069fa9ed42be6f8ccd75ed9d" exitCode=0
Jul 24 12:19:50.761163 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:50.761118    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bk4w6" event={"ID":"f42ba11d-0506-4915-bc77-d52d5fd3ea4a","Type":"ContainerDied","Data":"1a72984584af6f0bb24923ea549f6ead47fe5c50069fa9ed42be6f8ccd75ed9d"}
Jul 24 12:19:50.762754 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:50.762733    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sj8p9" event={"ID":"c74cea6b-83f7-4615-918a-d27808ed55ee","Type":"ContainerStarted","Data":"4cec528aa4571551d66b28308969757cd75bb2e6032ad63d06e40f872c43a8bd"}
Jul 24 12:19:50.768824 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:50.768768    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tvdgf" podStartSLOduration=3.921580625 podStartE2EDuration="22.768750368s" totalImagesPullingTime="18.847169743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:50.768009926 +0000 UTC m=+22.723014688" watchObservedRunningTime="2026-07-24 12:19:50.768750368 +0000 UTC m=+22.723755132"
Jul 24 12:19:50.769009 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:50.768972    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-194.ec2.internal" podStartSLOduration=21.768962865 podStartE2EDuration="21.768962865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:33.734398081 +0000 UTC m=+5.689402844" watchObservedRunningTime="2026-07-24 12:19:50.768962865 +0000 UTC m=+22.723967630"
Jul 24 12:19:50.780756 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:50.780715    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hrsjh" podStartSLOduration=4.390156958 podStartE2EDuration="22.780699796s" totalImagesPullingTime="18.390542838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:50.780234731 +0000 UTC m=+22.735239505" watchObservedRunningTime="2026-07-24 12:19:50.780699796 +0000 UTC m=+22.735704558"
Jul 24 12:19:50.794107 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:50.794055    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jmmcs" podStartSLOduration=3.9439469799999998 podStartE2EDuration="22.794035659s" totalImagesPullingTime="18.850088679s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:50.793521681 +0000 UTC m=+22.748526518" watchObservedRunningTime="2026-07-24 12:19:50.794035659 +0000 UTC m=+22.749040428"
Jul 24 12:19:50.811882 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:50.811836    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jzmtg" podStartSLOduration=3.9194714189999997 podStartE2EDuration="22.811820177s" totalImagesPullingTime="18.892348758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:50.811756442 +0000 UTC m=+22.766761205" watchObservedRunningTime="2026-07-24 12:19:50.811820177 +0000 UTC m=+22.766824939"
Jul 24 12:19:50.843233 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:50.843146    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sj8p9" podStartSLOduration=4.027350168 podStartE2EDuration="22.843127663s" totalImagesPullingTime="18.815777495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:50.842590478 +0000 UTC m=+22.797595238" watchObservedRunningTime="2026-07-24 12:19:50.843127663 +0000 UTC m=+22.798132426"
Jul 24 12:19:51.768579 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:51.768491    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k" event={"ID":"45d555da-de43-4a10-85d3-b24a77d99118","Type":"ContainerStarted","Data":"9624fa9398ba1704792f46eaefd5724914898fed6efd1014f3e3e992e3debdab"}
Jul 24 12:19:51.769232 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:51.768636    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k" event={"ID":"45d555da-de43-4a10-85d3-b24a77d99118","Type":"ContainerStarted","Data":"80d6a53d602647bd320350fb5638e5f7796c4c07cce80f56ca8eb49382118f03"}
Jul 24 12:19:51.769232 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:51.768693    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k" event={"ID":"45d555da-de43-4a10-85d3-b24a77d99118","Type":"ContainerStarted","Data":"adbf925a5ce013e7ed48b355753aaf1da4ff4365fcdb46b2d2ffdf4761abeb24"}
Jul 24 12:19:51.769232 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:51.768744    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k" event={"ID":"45d555da-de43-4a10-85d3-b24a77d99118","Type":"ContainerStarted","Data":"524e6333b7a0057ee963aa91dc616f6cbc8a18827a4cb01cc05825e436e621e2"}
Jul 24 12:19:51.769232 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:51.768790    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k" event={"ID":"45d555da-de43-4a10-85d3-b24a77d99118","Type":"ContainerStarted","Data":"ff2542d0630f2ce434a6692235373281a146da193c07522ed489bf6ea5f139b2"}
Jul 24 12:19:52.088623 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:52.088587    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 12:19:52.520370 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:52.520254    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T12:19:52.088615475Z","UUID":"6b3d7698-bc40-4505-8e8d-afc7f94c6845","Handler":null,"Name":"","Endpoint":""}
Jul 24 12:19:52.522070 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:52.522039    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 12:19:52.522206 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:52.522086    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 12:19:52.640064 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:52.639900    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:52.640064 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:52.639942    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:52.640064 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:52.639900    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:52.640064 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:52.640022    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrzvj" podUID="d98f89d0-11ef-4c6b-b973-64267283f1da"
Jul 24 12:19:52.640340 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:52.640111    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8294s" podUID="edc2499b-10da-416b-831d-52f00cbd026e"
Jul 24 12:19:52.640340 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:52.640194    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtb2z" podUID="bda12f0d-5325-4258-a601-381a0751907a"
Jul 24 12:19:52.772943 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:52.772860    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z" event={"ID":"138735f1-3e23-48e4-88a5-7992d2987d07","Type":"ContainerStarted","Data":"70c560260d9fd66144630a9a4aac0d007b2f85d902faa800d8377f20dcf88a46"}
Jul 24 12:19:52.774304 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:52.774272    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k6gfj" event={"ID":"1659a0ea-58c5-41f5-b8a6-b59ac3ae0ef3","Type":"ContainerStarted","Data":"66118e1c43b5db35f3d37151ca754e3a98e162db5fa9dbb952d61b1175abbcb5"}
Jul 24 12:19:52.790104 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:52.790052    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k6gfj" podStartSLOduration=4.974304242 podStartE2EDuration="23.790034362s" totalImagesPullingTime="18.81573012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:52.789982329 +0000 UTC m=+24.744987117" watchObservedRunningTime="2026-07-24 12:19:52.790034362 +0000 UTC m=+24.745039125"
Jul 24 12:19:53.784794 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:53.784759    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k" event={"ID":"45d555da-de43-4a10-85d3-b24a77d99118","Type":"ContainerStarted","Data":"a878824c0a239ac68954280220291ef25106c2ba55600432629004b1271583bd"}
Jul 24 12:19:54.639397 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:54.639360    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:54.639606 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:54.639362    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:54.639606 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:54.639523    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtb2z" podUID="bda12f0d-5325-4258-a601-381a0751907a"
Jul 24 12:19:54.639606 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:54.639582    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrzvj" podUID="d98f89d0-11ef-4c6b-b973-64267283f1da"
Jul 24 12:19:54.639606 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:54.639362    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:54.639827 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:54.639673    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8294s" podUID="edc2499b-10da-416b-831d-52f00cbd026e"
Jul 24 12:19:55.155752 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:55.155567    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hrsjh"
Jul 24 12:19:55.156203 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:55.156076    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hrsjh"
Jul 24 12:19:55.791095 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:55.791066    2581 generic.go:356] "Generic (PLEG): container finished" podID="f42ba11d-0506-4915-bc77-d52d5fd3ea4a" containerID="ce6ab926c5832269bc7f4569c7a00dd313a6e44cf1613c9c723f2e70b2649447" exitCode=0
Jul 24 12:19:55.791309 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:55.791146    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bk4w6" event={"ID":"f42ba11d-0506-4915-bc77-d52d5fd3ea4a","Type":"ContainerDied","Data":"ce6ab926c5832269bc7f4569c7a00dd313a6e44cf1613c9c723f2e70b2649447"}
Jul 24 12:19:55.793398 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:55.793371    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z" event={"ID":"138735f1-3e23-48e4-88a5-7992d2987d07","Type":"ContainerStarted","Data":"e721827b8ee6f687d344e049bdc82f4fad3d71d52baf337834798ccd723bb913"}
Jul 24 12:19:55.799589 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:55.799569    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k" event={"ID":"45d555da-de43-4a10-85d3-b24a77d99118","Type":"ContainerStarted","Data":"30daf12d82775c27c774e83288a4c964ca7b2d8c364138b1793aae976b4244d8"}
Jul 24 12:19:55.799818 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:55.799802    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hrsjh"
Jul 24 12:19:55.800220 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:55.800205    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hrsjh"
Jul 24 12:19:55.842822 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:55.842738    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sp62z" podStartSLOduration=5.206448497 podStartE2EDuration="27.842722291s" totalImagesPullingTime="22.636273794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:19:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:55.842668205 +0000 UTC m=+27.797672979" watchObservedRunningTime="2026-07-24 12:19:55.842722291 +0000 UTC m=+27.797727054"
Jul 24 12:19:55.865547 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:55.865491    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k" podStartSLOduration=8.957149904 podStartE2EDuration="27.865476025s" totalImagesPullingTime="18.908326121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:19:55.8646676 +0000 UTC m=+27.819672363" watchObservedRunningTime="2026-07-24 12:19:55.865476025 +0000 UTC m=+27.820480787"
Jul 24 12:19:56.640120 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:56.639912    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:56.640854 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:56.639912    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:56.640854 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:56.640205    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtb2z" podUID="bda12f0d-5325-4258-a601-381a0751907a"
Jul 24 12:19:56.640854 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:56.640001    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:56.640854 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:56.640270    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8294s" podUID="edc2499b-10da-416b-831d-52f00cbd026e"
Jul 24 12:19:56.640854 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:56.640364    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrzvj" podUID="d98f89d0-11ef-4c6b-b973-64267283f1da"
Jul 24 12:19:56.806588 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:56.806549    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:56.806796 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:56.806775    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:56.806912 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:56.806844    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:56.823361 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:56.823334    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:56.823511 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:56.823490    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:19:57.805641 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:57.805605    2581 generic.go:356] "Generic (PLEG): container finished" podID="f42ba11d-0506-4915-bc77-d52d5fd3ea4a" containerID="aed823fe628f14a39329a0ef4c3475290fcc91aa0fbe588761679ed0ac3498a5" exitCode=0
Jul 24 12:19:57.806107 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:57.805677    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bk4w6" event={"ID":"f42ba11d-0506-4915-bc77-d52d5fd3ea4a","Type":"ContainerDied","Data":"aed823fe628f14a39329a0ef4c3475290fcc91aa0fbe588761679ed0ac3498a5"}
Jul 24 12:19:58.643253 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:58.643224    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:19:58.643429 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:58.643318    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:19:58.643429 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:58.643326    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrzvj" podUID="d98f89d0-11ef-4c6b-b973-64267283f1da"
Jul 24 12:19:58.643429 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:58.643343    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:19:58.643550 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:58.643437    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtb2z" podUID="bda12f0d-5325-4258-a601-381a0751907a"
Jul 24 12:19:58.643550 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:19:58.643498    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8294s" podUID="edc2499b-10da-416b-831d-52f00cbd026e"
Jul 24 12:19:59.813214 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:59.813179    2581 generic.go:356] "Generic (PLEG): container finished" podID="f42ba11d-0506-4915-bc77-d52d5fd3ea4a" containerID="ef8b9947b81cd7d633fb269c3ac71206149072fc907e602da887028228d0223f" exitCode=0
Jul 24 12:19:59.813743 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:19:59.813256    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bk4w6" event={"ID":"f42ba11d-0506-4915-bc77-d52d5fd3ea4a","Type":"ContainerDied","Data":"ef8b9947b81cd7d633fb269c3ac71206149072fc907e602da887028228d0223f"}
Jul 24 12:20:00.639604 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:00.639566    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:20:00.639795 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:00.639580    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:20:00.639795 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:00.639705    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:20:00.639795 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:00.639701    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtb2z" podUID="bda12f0d-5325-4258-a601-381a0751907a"
Jul 24 12:20:00.639946 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:00.639809    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8294s" podUID="edc2499b-10da-416b-831d-52f00cbd026e"
Jul 24 12:20:00.639946 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:00.639863    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrzvj" podUID="d98f89d0-11ef-4c6b-b973-64267283f1da"
Jul 24 12:20:02.230299 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:02.230258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs\") pod \"network-metrics-daemon-wtb2z\" (UID: \"bda12f0d-5325-4258-a601-381a0751907a\") " pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:20:02.230734 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:02.230380    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:20:02.230734 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:02.230466    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs podName:bda12f0d-5325-4258-a601-381a0751907a nodeName:}" failed. No retries permitted until 2026-07-24 12:20:34.2304484 +0000 UTC m=+66.185453154 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs") pod "network-metrics-daemon-wtb2z" (UID: "bda12f0d-5325-4258-a601-381a0751907a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 12:20:02.330640 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:02.330596    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret\") pod \"global-pull-secret-syncer-qrzvj\" (UID: \"d98f89d0-11ef-4c6b-b973-64267283f1da\") " pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:20:02.330823 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:02.330659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm2k2\" (UniqueName: \"kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2\") pod \"network-check-target-8294s\" (UID: \"edc2499b-10da-416b-831d-52f00cbd026e\") " pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:20:02.330823 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:02.330780    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 12:20:02.330934 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:02.330851    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret podName:d98f89d0-11ef-4c6b-b973-64267283f1da nodeName:}" failed. No retries permitted until 2026-07-24 12:20:34.330835203 +0000 UTC m=+66.285839946 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret") pod "global-pull-secret-syncer-qrzvj" (UID: "d98f89d0-11ef-4c6b-b973-64267283f1da") : object "kube-system"/"original-pull-secret" not registered
Jul 24 12:20:02.330934 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:02.330785    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 12:20:02.330934 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:02.330902    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 12:20:02.330934 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:02.330916    2581 projected.go:196] Error preparing data for projected volume kube-api-access-bm2k2 for pod openshift-network-diagnostics/network-check-target-8294s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:20:02.331075 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:02.330962    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2 podName:edc2499b-10da-416b-831d-52f00cbd026e nodeName:}" failed. No retries permitted until 2026-07-24 12:20:34.330950063 +0000 UTC m=+66.285954822 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bm2k2" (UniqueName: "kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2") pod "network-check-target-8294s" (UID: "edc2499b-10da-416b-831d-52f00cbd026e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 12:20:02.639955 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:02.639914    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:20:02.640148 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:02.640049    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrzvj" podUID="d98f89d0-11ef-4c6b-b973-64267283f1da"
Jul 24 12:20:02.640476 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:02.640452    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:20:02.640622 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:02.640557    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtb2z" podUID="bda12f0d-5325-4258-a601-381a0751907a"
Jul 24 12:20:02.640622 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:02.640615    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:20:02.640742 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:02.640682    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8294s" podUID="edc2499b-10da-416b-831d-52f00cbd026e"
Jul 24 12:20:04.601867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:04.601624    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qrzvj"]
Jul 24 12:20:04.602209 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:04.601916    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:20:04.602209 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:04.602004    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrzvj" podUID="d98f89d0-11ef-4c6b-b973-64267283f1da"
Jul 24 12:20:04.608131 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:04.608099    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8294s"]
Jul 24 12:20:04.608265 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:04.608195    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:20:04.608308 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:04.608265    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8294s" podUID="edc2499b-10da-416b-831d-52f00cbd026e"
Jul 24 12:20:04.612146 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:04.612117    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wtb2z"]
Jul 24 12:20:04.612286 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:04.612250    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:20:04.612390 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:04.612364    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtb2z" podUID="bda12f0d-5325-4258-a601-381a0751907a"
Jul 24 12:20:04.836235 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:04.836205    2581 generic.go:356] "Generic (PLEG): container finished" podID="f42ba11d-0506-4915-bc77-d52d5fd3ea4a" containerID="5b8fbdde4ff86c5956cd7d1dcb4ae48d510305f180c06f33404c7aeb4c91362b" exitCode=0
Jul 24 12:20:04.836384 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:04.836253    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bk4w6" event={"ID":"f42ba11d-0506-4915-bc77-d52d5fd3ea4a","Type":"ContainerDied","Data":"5b8fbdde4ff86c5956cd7d1dcb4ae48d510305f180c06f33404c7aeb4c91362b"}
Jul 24 12:20:05.841519 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:05.841469    2581 generic.go:356] "Generic (PLEG): container finished" podID="f42ba11d-0506-4915-bc77-d52d5fd3ea4a" containerID="d66c7698ebebd7be39f74adca37c42b325612a1a172d79b1739b8c8a25525a20" exitCode=0
Jul 24 12:20:05.842065 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:05.841536    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bk4w6" event={"ID":"f42ba11d-0506-4915-bc77-d52d5fd3ea4a","Type":"ContainerDied","Data":"d66c7698ebebd7be39f74adca37c42b325612a1a172d79b1739b8c8a25525a20"}
Jul 24 12:20:06.639458 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:06.639403    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:20:06.639628 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:06.639403    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:20:06.639628 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:06.639530    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8294s" podUID="edc2499b-10da-416b-831d-52f00cbd026e"
Jul 24 12:20:06.639770 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:06.639635    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wtb2z" podUID="bda12f0d-5325-4258-a601-381a0751907a"
Jul 24 12:20:06.639770 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:06.639438    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:20:06.639770 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:06.639723    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrzvj" podUID="d98f89d0-11ef-4c6b-b973-64267283f1da"
Jul 24 12:20:06.847288 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:06.847258    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bk4w6" event={"ID":"f42ba11d-0506-4915-bc77-d52d5fd3ea4a","Type":"ContainerStarted","Data":"e0c21b9fc57714a749b8db8fb0d373e7cad84cf315eb43b5c7f68f2b9ad02e5d"}
Jul 24 12:20:06.874244 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:06.874196    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bk4w6" podStartSLOduration=9.200083479 podStartE2EDuration="38.874180002s" totalImagesPullingTime="29.674096523s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:19:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:06.873291179 +0000 UTC m=+38.828295944" watchObservedRunningTime="2026-07-24 12:20:06.874180002 +0000 UTC m=+38.829184765"
Jul 24 12:20:08.397961 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.397933    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-194.ec2.internal" event="NodeReady"
Jul 24 12:20:08.398330 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.398066    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 12:20:08.443397 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.443359    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8dqxn"]
Jul 24 12:20:08.476734 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.476701    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-564ct"]
Jul 24 12:20:08.476895 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.476832    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8dqxn"
Jul 24 12:20:08.478744 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.478723    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 12:20:08.479011 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.478996    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-68vm8\""
Jul 24 12:20:08.479259 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.479244    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 12:20:08.490553 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.490534    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8dqxn"]
Jul 24 12:20:08.490630 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.490565    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-564ct"]
Jul 24 12:20:08.490670 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.490635    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-564ct"
Jul 24 12:20:08.492633 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.492616    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 12:20:08.492728 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.492715    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kcvkr\""
Jul 24 12:20:08.492785 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.492753    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 12:20:08.492785 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.492761    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 12:20:08.549510 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.549481    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rzcwk"]
Jul 24 12:20:08.570433 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.570394    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rzcwk"]
Jul 24 12:20:08.570553 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.570527    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rzcwk"
Jul 24 12:20:08.572767 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.572745    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 12:20:08.572904 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.572804    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 12:20:08.572904 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.572749    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sd4k9\""
Jul 24 12:20:08.572904 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.572848    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 12:20:08.573047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.572748    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 12:20:08.574427 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.574389    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef8ade5f-a030-41c8-b643-6f9ef781a148-tmp-dir\") pod \"dns-default-8dqxn\" (UID: \"ef8ade5f-a030-41c8-b643-6f9ef781a148\") " pod="openshift-dns/dns-default-8dqxn"
Jul 24 12:20:08.574523 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.574457    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef8ade5f-a030-41c8-b643-6f9ef781a148-config-volume\") pod \"dns-default-8dqxn\" (UID: \"ef8ade5f-a030-41c8-b643-6f9ef781a148\") " pod="openshift-dns/dns-default-8dqxn"
Jul 24 12:20:08.574523 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.574519    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef8ade5f-a030-41c8-b643-6f9ef781a148-metrics-tls\") pod \"dns-default-8dqxn\" (UID: \"ef8ade5f-a030-41c8-b643-6f9ef781a148\") " pod="openshift-dns/dns-default-8dqxn"
Jul 24 12:20:08.574613 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.574549    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7h69\" (UniqueName: \"kubernetes.io/projected/ef8ade5f-a030-41c8-b643-6f9ef781a148-kube-api-access-q7h69\") pod \"dns-default-8dqxn\" (UID: \"ef8ade5f-a030-41c8-b643-6f9ef781a148\") " pod="openshift-dns/dns-default-8dqxn"
Jul 24 12:20:08.640586 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.640548    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:20:08.640754 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.640629    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:20:08.640837 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.640822    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:20:08.642973 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.642928    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 12:20:08.642973 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.642929    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zhxrp\""
Jul 24 12:20:08.642973 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.642931    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 12:20:08.643247 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.643005    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k4xp5\""
Jul 24 12:20:08.643309 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.643260    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 12:20:08.643384 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.643340    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 12:20:08.675886 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.675850    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zf57\" (UniqueName: \"kubernetes.io/projected/aa4a90db-e263-4e0d-ad83-9b0a60b282a0-kube-api-access-5zf57\") pod \"insights-runtime-extractor-rzcwk\" (UID: \"aa4a90db-e263-4e0d-ad83-9b0a60b282a0\") " pod="openshift-insights/insights-runtime-extractor-rzcwk"
Jul 24 12:20:08.676034 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.675897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aa4a90db-e263-4e0d-ad83-9b0a60b282a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzcwk\" (UID: \"aa4a90db-e263-4e0d-ad83-9b0a60b282a0\") " pod="openshift-insights/insights-runtime-extractor-rzcwk"
Jul 24 12:20:08.676034 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.675929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef8ade5f-a030-41c8-b643-6f9ef781a148-metrics-tls\") pod \"dns-default-8dqxn\" (UID: \"ef8ade5f-a030-41c8-b643-6f9ef781a148\") " pod="openshift-dns/dns-default-8dqxn"
Jul 24 12:20:08.676034 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.675954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7h69\" (UniqueName: \"kubernetes.io/projected/ef8ade5f-a030-41c8-b643-6f9ef781a148-kube-api-access-q7h69\") pod \"dns-default-8dqxn\" (UID: \"ef8ade5f-a030-41c8-b643-6f9ef781a148\") " pod="openshift-dns/dns-default-8dqxn"
Jul 24 12:20:08.676034 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.675983    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dbc2bb0f-05f3-4758-bcc8-71245bf72991-cert\") pod \"ingress-canary-564ct\" (UID: \"dbc2bb0f-05f3-4758-bcc8-71245bf72991\") " pod="openshift-ingress-canary/ingress-canary-564ct"
Jul 24 12:20:08.676153 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.676065    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef8ade5f-a030-41c8-b643-6f9ef781a148-tmp-dir\") pod \"dns-default-8dqxn\" (UID: \"ef8ade5f-a030-41c8-b643-6f9ef781a148\") " pod="openshift-dns/dns-default-8dqxn"
Jul 24 12:20:08.676153 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.676105    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvhgc\" (UniqueName: \"kubernetes.io/projected/dbc2bb0f-05f3-4758-bcc8-71245bf72991-kube-api-access-qvhgc\") pod \"ingress-canary-564ct\" (UID: \"dbc2bb0f-05f3-4758-bcc8-71245bf72991\") " pod="openshift-ingress-canary/ingress-canary-564ct"
Jul 24 12:20:08.676153 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.676133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aa4a90db-e263-4e0d-ad83-9b0a60b282a0-crio-socket\") pod \"insights-runtime-extractor-rzcwk\" (UID: \"aa4a90db-e263-4e0d-ad83-9b0a60b282a0\") " pod="openshift-insights/insights-runtime-extractor-rzcwk"
Jul 24 12:20:08.676258 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.676156    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aa4a90db-e263-4e0d-ad83-9b0a60b282a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzcwk\" (UID: \"aa4a90db-e263-4e0d-ad83-9b0a60b282a0\") " pod="openshift-insights/insights-runtime-extractor-rzcwk"
Jul 24 12:20:08.676258 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.676190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef8ade5f-a030-41c8-b643-6f9ef781a148-config-volume\") pod \"dns-default-8dqxn\" (UID: \"ef8ade5f-a030-41c8-b643-6f9ef781a148\") " pod="openshift-dns/dns-default-8dqxn"
Jul 24 12:20:08.676398 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.676379    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa4a90db-e263-4e0d-ad83-9b0a60b282a0-data-volume\") pod \"insights-runtime-extractor-rzcwk\" (UID: \"aa4a90db-e263-4e0d-ad83-9b0a60b282a0\") " pod="openshift-insights/insights-runtime-extractor-rzcwk"
Jul 24 12:20:08.676485 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.676440    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef8ade5f-a030-41c8-b643-6f9ef781a148-tmp-dir\") pod \"dns-default-8dqxn\" (UID: \"ef8ade5f-a030-41c8-b643-6f9ef781a148\") " pod="openshift-dns/dns-default-8dqxn"
Jul 24 12:20:08.676778 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.676756    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef8ade5f-a030-41c8-b643-6f9ef781a148-config-volume\") pod \"dns-default-8dqxn\" (UID: \"ef8ade5f-a030-41c8-b643-6f9ef781a148\") " pod="openshift-dns/dns-default-8dqxn"
Jul 24 12:20:08.680303 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.680275    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef8ade5f-a030-41c8-b643-6f9ef781a148-metrics-tls\") pod \"dns-default-8dqxn\" (UID: \"ef8ade5f-a030-41c8-b643-6f9ef781a148\") " pod="openshift-dns/dns-default-8dqxn"
Jul 24 12:20:08.689677 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.689651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7h69\" (UniqueName: \"kubernetes.io/projected/ef8ade5f-a030-41c8-b643-6f9ef781a148-kube-api-access-q7h69\") pod \"dns-default-8dqxn\" (UID: \"ef8ade5f-a030-41c8-b643-6f9ef781a148\") " pod="openshift-dns/dns-default-8dqxn"
Jul 24 12:20:08.777714 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.777675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dbc2bb0f-05f3-4758-bcc8-71245bf72991-cert\") pod \"ingress-canary-564ct\" (UID: \"dbc2bb0f-05f3-4758-bcc8-71245bf72991\") " pod="openshift-ingress-canary/ingress-canary-564ct"
Jul 24 12:20:08.777896 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.777744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvhgc\" (UniqueName: \"kubernetes.io/projected/dbc2bb0f-05f3-4758-bcc8-71245bf72991-kube-api-access-qvhgc\") pod \"ingress-canary-564ct\" (UID: \"dbc2bb0f-05f3-4758-bcc8-71245bf72991\") " pod="openshift-ingress-canary/ingress-canary-564ct"
Jul 24 12:20:08.777896 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.777761    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aa4a90db-e263-4e0d-ad83-9b0a60b282a0-crio-socket\") pod \"insights-runtime-extractor-rzcwk\" (UID: \"aa4a90db-e263-4e0d-ad83-9b0a60b282a0\") " pod="openshift-insights/insights-runtime-extractor-rzcwk"
Jul 24 12:20:08.777896 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.777777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aa4a90db-e263-4e0d-ad83-9b0a60b282a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzcwk\" (UID: \"aa4a90db-e263-4e0d-ad83-9b0a60b282a0\") " pod="openshift-insights/insights-runtime-extractor-rzcwk"
Jul 24 12:20:08.780866 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.778187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa4a90db-e263-4e0d-ad83-9b0a60b282a0-data-volume\") pod \"insights-runtime-extractor-rzcwk\" (UID: \"aa4a90db-e263-4e0d-ad83-9b0a60b282a0\") " pod="openshift-insights/insights-runtime-extractor-rzcwk"
Jul 24 12:20:08.780866 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.778224    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aa4a90db-e263-4e0d-ad83-9b0a60b282a0-crio-socket\") pod \"insights-runtime-extractor-rzcwk\" (UID: \"aa4a90db-e263-4e0d-ad83-9b0a60b282a0\") " pod="openshift-insights/insights-runtime-extractor-rzcwk"
Jul 24 12:20:08.780866 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.778267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zf57\" (UniqueName: \"kubernetes.io/projected/aa4a90db-e263-4e0d-ad83-9b0a60b282a0-kube-api-access-5zf57\") pod \"insights-runtime-extractor-rzcwk\" (UID: \"aa4a90db-e263-4e0d-ad83-9b0a60b282a0\") " pod="openshift-insights/insights-runtime-extractor-rzcwk"
Jul 24 12:20:08.780866 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.778453    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aa4a90db-e263-4e0d-ad83-9b0a60b282a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzcwk\" (UID: \"aa4a90db-e263-4e0d-ad83-9b0a60b282a0\") " pod="openshift-insights/insights-runtime-extractor-rzcwk"
Jul 24 12:20:08.780866 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.778769    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa4a90db-e263-4e0d-ad83-9b0a60b282a0-data-volume\") pod \"insights-runtime-extractor-rzcwk\" (UID: \"aa4a90db-e263-4e0d-ad83-9b0a60b282a0\") " pod="openshift-insights/insights-runtime-extractor-rzcwk"
Jul 24 12:20:08.780866 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.778887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aa4a90db-e263-4e0d-ad83-9b0a60b282a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rzcwk\" (UID: \"aa4a90db-e263-4e0d-ad83-9b0a60b282a0\") " pod="openshift-insights/insights-runtime-extractor-rzcwk"
Jul 24 12:20:08.782592 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.782569    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aa4a90db-e263-4e0d-ad83-9b0a60b282a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rzcwk\" (UID: \"aa4a90db-e263-4e0d-ad83-9b0a60b282a0\") " pod="openshift-insights/insights-runtime-extractor-rzcwk"
Jul 24 12:20:08.782702 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.782684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dbc2bb0f-05f3-4758-bcc8-71245bf72991-cert\") pod \"ingress-canary-564ct\" (UID: \"dbc2bb0f-05f3-4758-bcc8-71245bf72991\") " pod="openshift-ingress-canary/ingress-canary-564ct"
Jul 24 12:20:08.786110 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.786086    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvhgc\" (UniqueName: \"kubernetes.io/projected/dbc2bb0f-05f3-4758-bcc8-71245bf72991-kube-api-access-qvhgc\") pod \"ingress-canary-564ct\" (UID: \"dbc2bb0f-05f3-4758-bcc8-71245bf72991\") " pod="openshift-ingress-canary/ingress-canary-564ct"
Jul 24 12:20:08.786214 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.786095    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zf57\" (UniqueName: \"kubernetes.io/projected/aa4a90db-e263-4e0d-ad83-9b0a60b282a0-kube-api-access-5zf57\") pod \"insights-runtime-extractor-rzcwk\" (UID: \"aa4a90db-e263-4e0d-ad83-9b0a60b282a0\") " pod="openshift-insights/insights-runtime-extractor-rzcwk"
Jul 24 12:20:08.786214 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.786199    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8dqxn"
Jul 24 12:20:08.799047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.799020    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-564ct"
Jul 24 12:20:08.879432 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:08.879376    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rzcwk"
Jul 24 12:20:09.027080 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.027048    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-564ct"]
Jul 24 12:20:09.030408 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:20:09.030368    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc2bb0f_05f3_4758_bcc8_71245bf72991.slice/crio-0d09ef56350b65b0059a8b548cdcede763eff2a5f78c09c75fbff20534e3d769 WatchSource:0}: Error finding container 0d09ef56350b65b0059a8b548cdcede763eff2a5f78c09c75fbff20534e3d769: Status 404 returned error can't find the container with id 0d09ef56350b65b0059a8b548cdcede763eff2a5f78c09c75fbff20534e3d769
Jul 24 12:20:09.038094 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.038064    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8dqxn"]
Jul 24 12:20:09.041574 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:20:09.041549    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8ade5f_a030_41c8_b643_6f9ef781a148.slice/crio-c761c384cfe0cd32ae684097bec6b60ca6fac7d45dd85bfc069ae500ab6ba971 WatchSource:0}: Error finding container c761c384cfe0cd32ae684097bec6b60ca6fac7d45dd85bfc069ae500ab6ba971: Status 404 returned error can't find the container with id c761c384cfe0cd32ae684097bec6b60ca6fac7d45dd85bfc069ae500ab6ba971
Jul 24 12:20:09.074497 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.074465    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rzcwk"]
Jul 24 12:20:09.077306 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:20:09.077278    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4a90db_e263_4e0d_ad83_9b0a60b282a0.slice/crio-65054d117840038e6e22495c5bcc2dbc7668a8e607f70d4225a949719b46fc06 WatchSource:0}: Error finding container 65054d117840038e6e22495c5bcc2dbc7668a8e607f70d4225a949719b46fc06: Status 404 returned error can't find the container with id 65054d117840038e6e22495c5bcc2dbc7668a8e607f70d4225a949719b46fc06
Jul 24 12:20:09.273528 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.273445    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-2ckvk"]
Jul 24 12:20:09.307654 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.307622    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-2ckvk"]
Jul 24 12:20:09.307829 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.307754    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-2ckvk"
Jul 24 12:20:09.310224 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.310200    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 12:20:09.310527 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.310499    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 24 12:20:09.310657 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.310559    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-75c9v\""
Jul 24 12:20:09.310657 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.310560    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 24 12:20:09.310657 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.310621    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 12:20:09.310850 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.310833    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 12:20:09.484867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.484835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knkhp\" (UniqueName: \"kubernetes.io/projected/91263e28-33ba-4bcf-854d-f3a614be63e2-kube-api-access-knkhp\") pod \"prometheus-operator-84d69f794d-2ckvk\" (UID: \"91263e28-33ba-4bcf-854d-f3a614be63e2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2ckvk"
Jul 24 12:20:09.484867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.484881    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91263e28-33ba-4bcf-854d-f3a614be63e2-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-2ckvk\" (UID: \"91263e28-33ba-4bcf-854d-f3a614be63e2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2ckvk"
Jul 24 12:20:09.485334 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.484910    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91263e28-33ba-4bcf-854d-f3a614be63e2-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-2ckvk\" (UID: \"91263e28-33ba-4bcf-854d-f3a614be63e2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2ckvk"
Jul 24 12:20:09.485334 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.485016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91263e28-33ba-4bcf-854d-f3a614be63e2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-2ckvk\" (UID: \"91263e28-33ba-4bcf-854d-f3a614be63e2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2ckvk"
Jul 24 12:20:09.585881 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.585845    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91263e28-33ba-4bcf-854d-f3a614be63e2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-2ckvk\" (UID: \"91263e28-33ba-4bcf-854d-f3a614be63e2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2ckvk"
Jul 24 12:20:09.586067 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.585913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knkhp\" (UniqueName: \"kubernetes.io/projected/91263e28-33ba-4bcf-854d-f3a614be63e2-kube-api-access-knkhp\") pod \"prometheus-operator-84d69f794d-2ckvk\" (UID: \"91263e28-33ba-4bcf-854d-f3a614be63e2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2ckvk"
Jul 24 12:20:09.586067 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.585941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91263e28-33ba-4bcf-854d-f3a614be63e2-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-2ckvk\" (UID: \"91263e28-33ba-4bcf-854d-f3a614be63e2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2ckvk"
Jul 24 12:20:09.586067 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.585962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91263e28-33ba-4bcf-854d-f3a614be63e2-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-2ckvk\" (UID: \"91263e28-33ba-4bcf-854d-f3a614be63e2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2ckvk"
Jul 24 12:20:09.586819 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.586773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91263e28-33ba-4bcf-854d-f3a614be63e2-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-2ckvk\" (UID: \"91263e28-33ba-4bcf-854d-f3a614be63e2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2ckvk"
Jul 24 12:20:09.588860 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.588834    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91263e28-33ba-4bcf-854d-f3a614be63e2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-2ckvk\" (UID: \"91263e28-33ba-4bcf-854d-f3a614be63e2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2ckvk"
Jul 24 12:20:09.588860 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.588856    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91263e28-33ba-4bcf-854d-f3a614be63e2-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-2ckvk\" (UID: \"91263e28-33ba-4bcf-854d-f3a614be63e2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2ckvk"
Jul 24 12:20:09.594647 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.594612    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knkhp\" (UniqueName: \"kubernetes.io/projected/91263e28-33ba-4bcf-854d-f3a614be63e2-kube-api-access-knkhp\") pod \"prometheus-operator-84d69f794d-2ckvk\" (UID: \"91263e28-33ba-4bcf-854d-f3a614be63e2\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2ckvk"
Jul 24 12:20:09.617323 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.617287    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-2ckvk"
Jul 24 12:20:09.784455 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.784430    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-2ckvk"]
Jul 24 12:20:09.794929 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:20:09.794885    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91263e28_33ba_4bcf_854d_f3a614be63e2.slice/crio-48e7acc5cb3640d2df10bb61325cc92e9bdac432a13eea74def9555981279e9a WatchSource:0}: Error finding container 48e7acc5cb3640d2df10bb61325cc92e9bdac432a13eea74def9555981279e9a: Status 404 returned error can't find the container with id 48e7acc5cb3640d2df10bb61325cc92e9bdac432a13eea74def9555981279e9a
Jul 24 12:20:09.856706 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.856615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-2ckvk" event={"ID":"91263e28-33ba-4bcf-854d-f3a614be63e2","Type":"ContainerStarted","Data":"48e7acc5cb3640d2df10bb61325cc92e9bdac432a13eea74def9555981279e9a"}
Jul 24 12:20:09.858063 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.858034    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rzcwk" event={"ID":"aa4a90db-e263-4e0d-ad83-9b0a60b282a0","Type":"ContainerStarted","Data":"65054d117840038e6e22495c5bcc2dbc7668a8e607f70d4225a949719b46fc06"}
Jul 24 12:20:09.859377 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.859347    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-564ct" event={"ID":"dbc2bb0f-05f3-4758-bcc8-71245bf72991","Type":"ContainerStarted","Data":"0d09ef56350b65b0059a8b548cdcede763eff2a5f78c09c75fbff20534e3d769"}
Jul 24 12:20:09.860676 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:09.860654    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8dqxn" event={"ID":"ef8ade5f-a030-41c8-b643-6f9ef781a148","Type":"ContainerStarted","Data":"c761c384cfe0cd32ae684097bec6b60ca6fac7d45dd85bfc069ae500ab6ba971"}
Jul 24 12:20:14.682237 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:14.682205    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55ff44d7df-rgr25"]
Jul 24 12:20:14.893632 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:14.893596    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55ff44d7df-rgr25"]
Jul 24 12:20:14.893799 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:14.893732    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:14.896818 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:14.896791    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 24 12:20:14.896967 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:14.896939    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 24 12:20:14.897620 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:14.897595    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 24 12:20:14.897620 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:14.897612    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-l6rpx\""
Jul 24 12:20:14.897775 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:14.897677    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 24 12:20:14.897775 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:14.897681    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 24 12:20:14.897775 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:14.897746    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 24 12:20:14.897775 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:14.897756    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 24 12:20:14.902599 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:14.902579    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 24 12:20:14.927498 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:14.927467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-trusted-ca-bundle\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:14.927655 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:14.927538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-service-ca\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:14.927655 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:14.927608    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-oauth-serving-cert\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:14.927765 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:14.927660    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5750adc2-c7a4-466b-a73a-662a71446c4a-console-serving-cert\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:14.927765 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:14.927683    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-console-config\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:14.927765 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:14.927734    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rktl\" (UniqueName: \"kubernetes.io/projected/5750adc2-c7a4-466b-a73a-662a71446c4a-kube-api-access-4rktl\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:14.927765 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:14.927760    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5750adc2-c7a4-466b-a73a-662a71446c4a-console-oauth-config\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:15.028389 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:15.028296    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-trusted-ca-bundle\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:15.028568 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:15.028395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-service-ca\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:15.028568 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:15.028461    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-oauth-serving-cert\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:15.028568 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:15.028522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5750adc2-c7a4-466b-a73a-662a71446c4a-console-serving-cert\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:15.028568 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:15.028549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-console-config\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:15.028809 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:15.028589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rktl\" (UniqueName: \"kubernetes.io/projected/5750adc2-c7a4-466b-a73a-662a71446c4a-kube-api-access-4rktl\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:15.028809 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:15.028629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5750adc2-c7a4-466b-a73a-662a71446c4a-console-oauth-config\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:15.029259 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:15.029207    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-service-ca\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:15.029476 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:15.029456    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-trusted-ca-bundle\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:15.029646 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:15.029622    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-console-config\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:15.029724 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:15.029701    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-oauth-serving-cert\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:15.031393 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:15.031373    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5750adc2-c7a4-466b-a73a-662a71446c4a-console-oauth-config\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:15.031805 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:15.031787    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5750adc2-c7a4-466b-a73a-662a71446c4a-console-serving-cert\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:15.036958 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:15.036926    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rktl\" (UniqueName: \"kubernetes.io/projected/5750adc2-c7a4-466b-a73a-662a71446c4a-kube-api-access-4rktl\") pod \"console-55ff44d7df-rgr25\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") " pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:15.207135 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:15.207097    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:16.446319 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:16.444954    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55ff44d7df-rgr25"]
Jul 24 12:20:16.514435 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:20:16.514371    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5750adc2_c7a4_466b_a73a_662a71446c4a.slice/crio-2dbc1bf196680adf672039bb0f65228bcfb6221a98e55b3fa592a79680fedd71 WatchSource:0}: Error finding container 2dbc1bf196680adf672039bb0f65228bcfb6221a98e55b3fa592a79680fedd71: Status 404 returned error can't find the container with id 2dbc1bf196680adf672039bb0f65228bcfb6221a98e55b3fa592a79680fedd71
Jul 24 12:20:16.883113 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:16.883005    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-564ct" event={"ID":"dbc2bb0f-05f3-4758-bcc8-71245bf72991","Type":"ContainerStarted","Data":"c8ff50d3b6487b29e92f3c15c36e18cb5caa357d03aee44f1aeb1e4bf1582039"}
Jul 24 12:20:16.884813 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:16.884786    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8dqxn" event={"ID":"ef8ade5f-a030-41c8-b643-6f9ef781a148","Type":"ContainerStarted","Data":"0a337892dcd85f7577bf5392fb2bd41069b905841cbad277c4e8f51c7224ce1f"}
Jul 24 12:20:16.884938 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:16.884832    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8dqxn" event={"ID":"ef8ade5f-a030-41c8-b643-6f9ef781a148","Type":"ContainerStarted","Data":"2ee571ec19e62666aa431027c725ddc754d9a5b0abe64ac353dc87cad7a321fd"}
Jul 24 12:20:16.884938 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:16.884860    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8dqxn"
Jul 24 12:20:16.886597 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:16.886574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-2ckvk" event={"ID":"91263e28-33ba-4bcf-854d-f3a614be63e2","Type":"ContainerStarted","Data":"29dea906fe62ef92958645af9af08624ff586b397b8c213f4e63b9b839e7dbf6"}
Jul 24 12:20:16.886697 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:16.886612    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-2ckvk" event={"ID":"91263e28-33ba-4bcf-854d-f3a614be63e2","Type":"ContainerStarted","Data":"b00464fd0c3cd7aa232e94c896d934778491a3dc28aa7b9067ce173415566e8a"}
Jul 24 12:20:16.887957 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:16.887934    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rzcwk" event={"ID":"aa4a90db-e263-4e0d-ad83-9b0a60b282a0","Type":"ContainerStarted","Data":"5a197388d1dae0de1eb182d76267fe8be6b82baa1f2fc54f983e6a0836b9ef45"}
Jul 24 12:20:16.889004 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:16.888966    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55ff44d7df-rgr25" event={"ID":"5750adc2-c7a4-466b-a73a-662a71446c4a","Type":"ContainerStarted","Data":"2dbc1bf196680adf672039bb0f65228bcfb6221a98e55b3fa592a79680fedd71"}
Jul 24 12:20:16.898372 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:16.898318    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-564ct" podStartSLOduration=1.629576525 podStartE2EDuration="8.898302403s" totalImagesPullingTime="7.268725878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:20:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:16.897924807 +0000 UTC m=+48.852929572" watchObservedRunningTime="2026-07-24 12:20:16.898302403 +0000 UTC m=+48.853307170"
Jul 24 12:20:16.915106 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:16.915059    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-2ckvk" podStartSLOduration=1.411013874 podStartE2EDuration="7.915040487s" totalImagesPullingTime="6.504026613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:20:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:16.914255891 +0000 UTC m=+48.869260677" watchObservedRunningTime="2026-07-24 12:20:16.915040487 +0000 UTC m=+48.870045229"
Jul 24 12:20:16.931070 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:16.930919    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8dqxn" podStartSLOduration=2.061845175 podStartE2EDuration="8.930902284s" totalImagesPullingTime="6.869057109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:20:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:16.930690464 +0000 UTC m=+48.885695228" watchObservedRunningTime="2026-07-24 12:20:16.930902284 +0000 UTC m=+48.885907051"
Jul 24 12:20:18.618148 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.618107    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5"]
Jul 24 12:20:18.643608 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.643575    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5"
Jul 24 12:20:18.646017 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.645994    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rg8n6\""
Jul 24 12:20:18.646186 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.646016    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 24 12:20:18.646298 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.646044    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 24 12:20:18.648127 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.648107    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5"]
Jul 24 12:20:18.648210 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.648170    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"]
Jul 24 12:20:18.660479 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.660403    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/215ec8d5-e824-4a7e-9b89-4b9ee4127682-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-q22x5\" (UID: \"215ec8d5-e824-4a7e-9b89-4b9ee4127682\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5"
Jul 24 12:20:18.660479 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.660471    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/215ec8d5-e824-4a7e-9b89-4b9ee4127682-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-q22x5\" (UID: \"215ec8d5-e824-4a7e-9b89-4b9ee4127682\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5"
Jul 24 12:20:18.660691 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.660532    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxlnp\" (UniqueName: \"kubernetes.io/projected/215ec8d5-e824-4a7e-9b89-4b9ee4127682-kube-api-access-pxlnp\") pod \"openshift-state-metrics-b8f49f5c6-q22x5\" (UID: \"215ec8d5-e824-4a7e-9b89-4b9ee4127682\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5"
Jul 24 12:20:18.660691 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.660667    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/215ec8d5-e824-4a7e-9b89-4b9ee4127682-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-q22x5\" (UID: \"215ec8d5-e824-4a7e-9b89-4b9ee4127682\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5"
Jul 24 12:20:18.665605 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.665558    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rrtq2"]
Jul 24 12:20:18.665760 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.665707    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:18.668080 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.668057    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 24 12:20:18.668200 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.668057    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mtjx2\""
Jul 24 12:20:18.668257 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.668122    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 24 12:20:18.668951 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.668570    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 24 12:20:18.674350 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.674330    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"]
Jul 24 12:20:18.674512 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.674495    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.676364 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.676334    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 12:20:18.676364 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.676341    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 12:20:18.676720 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.676699    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wc5gs\""
Jul 24 12:20:18.676824 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.676752    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 12:20:18.761567 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.761538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/215ec8d5-e824-4a7e-9b89-4b9ee4127682-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-q22x5\" (UID: \"215ec8d5-e824-4a7e-9b89-4b9ee4127682\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5"
Jul 24 12:20:18.761567 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.761576    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18cdccc7-5d25-4e5b-af0e-b0a40d851053-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-cbkfr\" (UID: \"18cdccc7-5d25-4e5b-af0e-b0a40d851053\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:18.761807 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.761609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxlnp\" (UniqueName: \"kubernetes.io/projected/215ec8d5-e824-4a7e-9b89-4b9ee4127682-kube-api-access-pxlnp\") pod \"openshift-state-metrics-b8f49f5c6-q22x5\" (UID: \"215ec8d5-e824-4a7e-9b89-4b9ee4127682\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5"
Jul 24 12:20:18.761807 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.761637    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chmzt\" (UniqueName: \"kubernetes.io/projected/99c008bb-acfe-480a-a07c-9bccdce3d34b-kube-api-access-chmzt\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.761807 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.761662    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99c008bb-acfe-480a-a07c-9bccdce3d34b-sys\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.761807 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.761690    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/18cdccc7-5d25-4e5b-af0e-b0a40d851053-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-cbkfr\" (UID: \"18cdccc7-5d25-4e5b-af0e-b0a40d851053\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:18.761807 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:18.761702    2581 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 24 12:20:18.761807 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.761717    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99c008bb-acfe-480a-a07c-9bccdce3d34b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.761807 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.761769    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28n7c\" (UniqueName: \"kubernetes.io/projected/18cdccc7-5d25-4e5b-af0e-b0a40d851053-kube-api-access-28n7c\") pod \"kube-state-metrics-56d9f944b4-cbkfr\" (UID: \"18cdccc7-5d25-4e5b-af0e-b0a40d851053\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:18.761807 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.761794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/215ec8d5-e824-4a7e-9b89-4b9ee4127682-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-q22x5\" (UID: \"215ec8d5-e824-4a7e-9b89-4b9ee4127682\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5"
Jul 24 12:20:18.762289 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.761822    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99c008bb-acfe-480a-a07c-9bccdce3d34b-node-exporter-textfile\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.762289 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.761842    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99c008bb-acfe-480a-a07c-9bccdce3d34b-metrics-client-ca\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.762289 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.761874    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99c008bb-acfe-480a-a07c-9bccdce3d34b-node-exporter-tls\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.762289 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.761955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99c008bb-acfe-480a-a07c-9bccdce3d34b-root\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.762289 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.761982    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99c008bb-acfe-480a-a07c-9bccdce3d34b-node-exporter-wtmp\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.762289 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:18.762023    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/215ec8d5-e824-4a7e-9b89-4b9ee4127682-openshift-state-metrics-tls podName:215ec8d5-e824-4a7e-9b89-4b9ee4127682 nodeName:}" failed. No retries permitted until 2026-07-24 12:20:19.261993171 +0000 UTC m=+51.216997915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/215ec8d5-e824-4a7e-9b89-4b9ee4127682-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-q22x5" (UID: "215ec8d5-e824-4a7e-9b89-4b9ee4127682") : secret "openshift-state-metrics-tls" not found
Jul 24 12:20:18.762289 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.762060    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/18cdccc7-5d25-4e5b-af0e-b0a40d851053-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-cbkfr\" (UID: \"18cdccc7-5d25-4e5b-af0e-b0a40d851053\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:18.762289 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.762105    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99c008bb-acfe-480a-a07c-9bccdce3d34b-node-exporter-accelerators-collector-config\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.762289 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.762129    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/18cdccc7-5d25-4e5b-af0e-b0a40d851053-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-cbkfr\" (UID: \"18cdccc7-5d25-4e5b-af0e-b0a40d851053\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:18.762289 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.762207    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/215ec8d5-e824-4a7e-9b89-4b9ee4127682-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-q22x5\" (UID: \"215ec8d5-e824-4a7e-9b89-4b9ee4127682\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5"
Jul 24 12:20:18.762289 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.762251    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18cdccc7-5d25-4e5b-af0e-b0a40d851053-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-cbkfr\" (UID: \"18cdccc7-5d25-4e5b-af0e-b0a40d851053\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:18.762732 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.762694    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/215ec8d5-e824-4a7e-9b89-4b9ee4127682-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-q22x5\" (UID: \"215ec8d5-e824-4a7e-9b89-4b9ee4127682\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5"
Jul 24 12:20:18.765035 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.765011    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/215ec8d5-e824-4a7e-9b89-4b9ee4127682-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-q22x5\" (UID: \"215ec8d5-e824-4a7e-9b89-4b9ee4127682\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5"
Jul 24 12:20:18.773781 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.773747    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxlnp\" (UniqueName: \"kubernetes.io/projected/215ec8d5-e824-4a7e-9b89-4b9ee4127682-kube-api-access-pxlnp\") pod \"openshift-state-metrics-b8f49f5c6-q22x5\" (UID: \"215ec8d5-e824-4a7e-9b89-4b9ee4127682\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5"
Jul 24 12:20:18.863709 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.863674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99c008bb-acfe-480a-a07c-9bccdce3d34b-node-exporter-textfile\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.863886 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.863719    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99c008bb-acfe-480a-a07c-9bccdce3d34b-metrics-client-ca\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.864089 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.864067    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99c008bb-acfe-480a-a07c-9bccdce3d34b-node-exporter-textfile\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.864151 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.864130    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99c008bb-acfe-480a-a07c-9bccdce3d34b-node-exporter-tls\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.864202 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.864173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99c008bb-acfe-480a-a07c-9bccdce3d34b-root\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.864202 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.864200    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99c008bb-acfe-480a-a07c-9bccdce3d34b-node-exporter-wtmp\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.864314 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.864245    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/18cdccc7-5d25-4e5b-af0e-b0a40d851053-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-cbkfr\" (UID: \"18cdccc7-5d25-4e5b-af0e-b0a40d851053\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:18.864314 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.864292    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99c008bb-acfe-480a-a07c-9bccdce3d34b-node-exporter-accelerators-collector-config\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.864393 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.864309    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99c008bb-acfe-480a-a07c-9bccdce3d34b-metrics-client-ca\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.864393 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.864313    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99c008bb-acfe-480a-a07c-9bccdce3d34b-root\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.864479 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:18.864406    2581 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 24 12:20:18.864479 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.864318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/18cdccc7-5d25-4e5b-af0e-b0a40d851053-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-cbkfr\" (UID: \"18cdccc7-5d25-4e5b-af0e-b0a40d851053\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:18.864479 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.864452    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99c008bb-acfe-480a-a07c-9bccdce3d34b-node-exporter-wtmp\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.864604 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:18.864487    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18cdccc7-5d25-4e5b-af0e-b0a40d851053-kube-state-metrics-tls podName:18cdccc7-5d25-4e5b-af0e-b0a40d851053 nodeName:}" failed. No retries permitted until 2026-07-24 12:20:19.364467562 +0000 UTC m=+51.319472305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/18cdccc7-5d25-4e5b-af0e-b0a40d851053-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-cbkfr" (UID: "18cdccc7-5d25-4e5b-af0e-b0a40d851053") : secret "kube-state-metrics-tls" not found
Jul 24 12:20:18.864604 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.864529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18cdccc7-5d25-4e5b-af0e-b0a40d851053-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-cbkfr\" (UID: \"18cdccc7-5d25-4e5b-af0e-b0a40d851053\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:18.864604 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.864594    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18cdccc7-5d25-4e5b-af0e-b0a40d851053-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-cbkfr\" (UID: \"18cdccc7-5d25-4e5b-af0e-b0a40d851053\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:18.864763 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.864636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chmzt\" (UniqueName: \"kubernetes.io/projected/99c008bb-acfe-480a-a07c-9bccdce3d34b-kube-api-access-chmzt\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.864763 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.864661    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99c008bb-acfe-480a-a07c-9bccdce3d34b-sys\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.864763 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.864691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/18cdccc7-5d25-4e5b-af0e-b0a40d851053-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-cbkfr\" (UID: \"18cdccc7-5d25-4e5b-af0e-b0a40d851053\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:18.864763 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.864723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99c008bb-acfe-480a-a07c-9bccdce3d34b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.864892 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.864775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28n7c\" (UniqueName: \"kubernetes.io/projected/18cdccc7-5d25-4e5b-af0e-b0a40d851053-kube-api-access-28n7c\") pod \"kube-state-metrics-56d9f944b4-cbkfr\" (UID: \"18cdccc7-5d25-4e5b-af0e-b0a40d851053\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:18.864986 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.864953    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99c008bb-acfe-480a-a07c-9bccdce3d34b-node-exporter-accelerators-collector-config\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.865055 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.865033    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/18cdccc7-5d25-4e5b-af0e-b0a40d851053-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-cbkfr\" (UID: \"18cdccc7-5d25-4e5b-af0e-b0a40d851053\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:18.865115 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.865084    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18cdccc7-5d25-4e5b-af0e-b0a40d851053-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-cbkfr\" (UID: \"18cdccc7-5d25-4e5b-af0e-b0a40d851053\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:18.865115 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.865085    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99c008bb-acfe-480a-a07c-9bccdce3d34b-sys\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.865251 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.865225    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/18cdccc7-5d25-4e5b-af0e-b0a40d851053-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-cbkfr\" (UID: \"18cdccc7-5d25-4e5b-af0e-b0a40d851053\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:18.867528 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.867505    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99c008bb-acfe-480a-a07c-9bccdce3d34b-node-exporter-tls\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.867615 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.867543    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99c008bb-acfe-480a-a07c-9bccdce3d34b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.872402 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.872333    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chmzt\" (UniqueName: \"kubernetes.io/projected/99c008bb-acfe-480a-a07c-9bccdce3d34b-kube-api-access-chmzt\") pod \"node-exporter-rrtq2\" (UID: \"99c008bb-acfe-480a-a07c-9bccdce3d34b\") " pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:18.872738 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.872590    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18cdccc7-5d25-4e5b-af0e-b0a40d851053-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-cbkfr\" (UID: \"18cdccc7-5d25-4e5b-af0e-b0a40d851053\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:18.872738 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.872687    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28n7c\" (UniqueName: \"kubernetes.io/projected/18cdccc7-5d25-4e5b-af0e-b0a40d851053-kube-api-access-28n7c\") pod \"kube-state-metrics-56d9f944b4-cbkfr\" (UID: \"18cdccc7-5d25-4e5b-af0e-b0a40d851053\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:18.986310 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:18.986276    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rrtq2"
Jul 24 12:20:19.269529 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.269444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/215ec8d5-e824-4a7e-9b89-4b9ee4127682-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-q22x5\" (UID: \"215ec8d5-e824-4a7e-9b89-4b9ee4127682\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5"
Jul 24 12:20:19.272176 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.272139    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/215ec8d5-e824-4a7e-9b89-4b9ee4127682-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-q22x5\" (UID: \"215ec8d5-e824-4a7e-9b89-4b9ee4127682\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5"
Jul 24 12:20:19.370135 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.370093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/18cdccc7-5d25-4e5b-af0e-b0a40d851053-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-cbkfr\" (UID: \"18cdccc7-5d25-4e5b-af0e-b0a40d851053\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:19.372931 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.372907    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/18cdccc7-5d25-4e5b-af0e-b0a40d851053-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-cbkfr\" (UID: \"18cdccc7-5d25-4e5b-af0e-b0a40d851053\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:19.555033 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.554994    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5"
Jul 24 12:20:19.576966 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.576938    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"
Jul 24 12:20:19.692793 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.692757    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 12:20:19.733674 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.733631    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 12:20:19.733834 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.733764    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.736232 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.736199    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 24 12:20:19.736232 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.736218    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 24 12:20:19.736232 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.736222    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 24 12:20:19.736484 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.736304    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 24 12:20:19.736484 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.736304    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dm8jn\""
Jul 24 12:20:19.736607 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.736585    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 24 12:20:19.736688 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.736588    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 24 12:20:19.736752 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.736737    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 24 12:20:19.737964 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.737941    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 24 12:20:19.738152 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.738131    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 24 12:20:19.775132 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.774467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54a7f8b4-055f-424b-87a5-d83333058107-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.775132 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.774511    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54a7f8b4-055f-424b-87a5-d83333058107-config-out\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.775132 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.774550    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.775132 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.774612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/54a7f8b4-055f-424b-87a5-d83333058107-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.775132 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.774636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54a7f8b4-055f-424b-87a5-d83333058107-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.775132 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.774666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-web-config\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.775132 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.774695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.775132 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.774734    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.775132 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.774763    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.775132 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.774788    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54a7f8b4-055f-424b-87a5-d83333058107-tls-assets\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.775132 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.774834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-config-volume\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.775132 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.774873    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q5nj\" (UniqueName: \"kubernetes.io/projected/54a7f8b4-055f-424b-87a5-d83333058107-kube-api-access-9q5nj\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.775132 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.774914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.875837 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.875756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/54a7f8b4-055f-424b-87a5-d83333058107-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.875837 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.875795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54a7f8b4-055f-424b-87a5-d83333058107-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.876061 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.875931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-web-config\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.876061 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.875992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.876061 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.876030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.876202 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.876067    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.876202 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.876098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54a7f8b4-055f-424b-87a5-d83333058107-tls-assets\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.876202 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.876152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-config-volume\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.876202 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.876178    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q5nj\" (UniqueName: \"kubernetes.io/projected/54a7f8b4-055f-424b-87a5-d83333058107-kube-api-access-9q5nj\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.876386 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.876207    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/54a7f8b4-055f-424b-87a5-d83333058107-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.876386 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.876219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.876386 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.876272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54a7f8b4-055f-424b-87a5-d83333058107-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.876386 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.876303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54a7f8b4-055f-424b-87a5-d83333058107-config-out\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.876386 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.876348    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.876679 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.876614    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54a7f8b4-055f-424b-87a5-d83333058107-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.877934 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:19.877100    2581 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 24 12:20:19.877934 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:19.877206    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-secret-alertmanager-main-tls podName:54a7f8b4-055f-424b-87a5-d83333058107 nodeName:}" failed. No retries permitted until 2026-07-24 12:20:20.377184922 +0000 UTC m=+52.332189667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "54a7f8b4-055f-424b-87a5-d83333058107") : secret "alertmanager-main-tls" not found
Jul 24 12:20:19.877934 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:19.877891    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/54a7f8b4-055f-424b-87a5-d83333058107-alertmanager-trusted-ca-bundle podName:54a7f8b4-055f-424b-87a5-d83333058107 nodeName:}" failed. No retries permitted until 2026-07-24 12:20:20.37787025 +0000 UTC m=+52.332875017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/54a7f8b4-055f-424b-87a5-d83333058107-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "54a7f8b4-055f-424b-87a5-d83333058107") : configmap references non-existent config key: ca-bundle.crt
Jul 24 12:20:19.879549 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.879502    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.880675 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.880652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.883549 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.880652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54a7f8b4-055f-424b-87a5-d83333058107-tls-assets\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.883549 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.880757    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.883549 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.881093    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-config-volume\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.883549 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.881349    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.883549 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.881918    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-web-config\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.883549 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.882630    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54a7f8b4-055f-424b-87a5-d83333058107-config-out\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:19.885670 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:19.885628    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q5nj\" (UniqueName: \"kubernetes.io/projected/54a7f8b4-055f-424b-87a5-d83333058107-kube-api-access-9q5nj\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:20.382403 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.382363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:20.382720 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.382468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54a7f8b4-055f-424b-87a5-d83333058107-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:20.383371 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.383340    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54a7f8b4-055f-424b-87a5-d83333058107-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:20.385242 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.385212    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/54a7f8b4-055f-424b-87a5-d83333058107-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"54a7f8b4-055f-424b-87a5-d83333058107\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:20.610308 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.610260    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"]
Jul 24 12:20:20.646309 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.646232    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 12:20:20.649971 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.649943    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.652240 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.652202    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 24 12:20:20.652516 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.652497    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 24 12:20:20.652651 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.652625    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 24 12:20:20.652757 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.652503    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-cl87f2bkj0ppr\""
Jul 24 12:20:20.652875 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.652859    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-j76r5\""
Jul 24 12:20:20.653290 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.653274    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 24 12:20:20.653606 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.653590    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 24 12:20:20.654585 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.654567    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"]
Jul 24 12:20:20.684885 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.684845    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/789a6a57-4d93-47d3-abd7-201f0c2d53eb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.685020 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.684894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/789a6a57-4d93-47d3-abd7-201f0c2d53eb-secret-grpc-tls\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.685020 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.684921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/789a6a57-4d93-47d3-abd7-201f0c2d53eb-metrics-client-ca\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.685020 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.684963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/789a6a57-4d93-47d3-abd7-201f0c2d53eb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.685153 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.685068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/789a6a57-4d93-47d3-abd7-201f0c2d53eb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.685153 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.685096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt2vr\" (UniqueName: \"kubernetes.io/projected/789a6a57-4d93-47d3-abd7-201f0c2d53eb-kube-api-access-pt2vr\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.685153 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.685129    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/789a6a57-4d93-47d3-abd7-201f0c2d53eb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.685266 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.685156    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/789a6a57-4d93-47d3-abd7-201f0c2d53eb-secret-thanos-querier-tls\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.786039 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.785994    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/789a6a57-4d93-47d3-abd7-201f0c2d53eb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.786529 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.786051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/789a6a57-4d93-47d3-abd7-201f0c2d53eb-secret-grpc-tls\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.786529 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.786088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/789a6a57-4d93-47d3-abd7-201f0c2d53eb-metrics-client-ca\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.786529 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.786138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/789a6a57-4d93-47d3-abd7-201f0c2d53eb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.786529 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.786214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/789a6a57-4d93-47d3-abd7-201f0c2d53eb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.786529 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.786242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt2vr\" (UniqueName: \"kubernetes.io/projected/789a6a57-4d93-47d3-abd7-201f0c2d53eb-kube-api-access-pt2vr\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.786529 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.786275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/789a6a57-4d93-47d3-abd7-201f0c2d53eb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.786529 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.786301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/789a6a57-4d93-47d3-abd7-201f0c2d53eb-secret-thanos-querier-tls\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.787269 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.787182    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/789a6a57-4d93-47d3-abd7-201f0c2d53eb-metrics-client-ca\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.789171 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.789137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/789a6a57-4d93-47d3-abd7-201f0c2d53eb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.789381 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.789289    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/789a6a57-4d93-47d3-abd7-201f0c2d53eb-secret-grpc-tls\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.789597 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.789573    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/789a6a57-4d93-47d3-abd7-201f0c2d53eb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.789678 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.789615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/789a6a57-4d93-47d3-abd7-201f0c2d53eb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.789772 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.789755    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/789a6a57-4d93-47d3-abd7-201f0c2d53eb-secret-thanos-querier-tls\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.790690 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.790643    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/789a6a57-4d93-47d3-abd7-201f0c2d53eb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.794277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.794235    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt2vr\" (UniqueName: \"kubernetes.io/projected/789a6a57-4d93-47d3-abd7-201f0c2d53eb-kube-api-access-pt2vr\") pod \"thanos-querier-85c5b8bcfb-bkqjk\" (UID: \"789a6a57-4d93-47d3-abd7-201f0c2d53eb\") " pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:20.962344 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:20.962255    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:22.527562 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:20:22.525482    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c008bb_acfe_480a_a07c_9bccdce3d34b.slice/crio-3e9319ed953d5e492a920d175c821fd99352401b14aa5e9bdfb03ecf89349952 WatchSource:0}: Error finding container 3e9319ed953d5e492a920d175c821fd99352401b14aa5e9bdfb03ecf89349952: Status 404 returned error can't find the container with id 3e9319ed953d5e492a920d175c821fd99352401b14aa5e9bdfb03ecf89349952
Jul 24 12:20:22.689208 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:22.685899    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 12:20:22.700929 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:22.700895    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr"]
Jul 24 12:20:22.704635 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:20:22.704606    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18cdccc7_5d25_4e5b_af0e_b0a40d851053.slice/crio-f49ad4b3a9386bbff4dd13bda21069186d7aad5e13a5565b86765fa87069d5de WatchSource:0}: Error finding container f49ad4b3a9386bbff4dd13bda21069186d7aad5e13a5565b86765fa87069d5de: Status 404 returned error can't find the container with id f49ad4b3a9386bbff4dd13bda21069186d7aad5e13a5565b86765fa87069d5de
Jul 24 12:20:22.720493 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:22.720392    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5"]
Jul 24 12:20:22.723599 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:20:22.723572    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215ec8d5_e824_4a7e_9b89_4b9ee4127682.slice/crio-eed3312888f7aab5446d825c947c0249b2130d7418c324b3a5010ae069c0e17e WatchSource:0}: Error finding container eed3312888f7aab5446d825c947c0249b2130d7418c324b3a5010ae069c0e17e: Status 404 returned error can't find the container with id eed3312888f7aab5446d825c947c0249b2130d7418c324b3a5010ae069c0e17e
Jul 24 12:20:22.727643 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:22.727585    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"]
Jul 24 12:20:22.729953 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:20:22.729932    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789a6a57_4d93_47d3_abd7_201f0c2d53eb.slice/crio-628f5507c9af596480746bce601c7fe4b441d5cc11880e9ee52c0b960370ae63 WatchSource:0}: Error finding container 628f5507c9af596480746bce601c7fe4b441d5cc11880e9ee52c0b960370ae63: Status 404 returned error can't find the container with id 628f5507c9af596480746bce601c7fe4b441d5cc11880e9ee52c0b960370ae63
Jul 24 12:20:22.916598 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:22.916561    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr" event={"ID":"18cdccc7-5d25-4e5b-af0e-b0a40d851053","Type":"ContainerStarted","Data":"f49ad4b3a9386bbff4dd13bda21069186d7aad5e13a5565b86765fa87069d5de"}
Jul 24 12:20:22.917966 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:22.917943    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55ff44d7df-rgr25" event={"ID":"5750adc2-c7a4-466b-a73a-662a71446c4a","Type":"ContainerStarted","Data":"342fdcebc5ffb1c9a6b224d48895a6128f534208e7c8c9dfcb61974f9aeccd97"}
Jul 24 12:20:22.918922 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:22.918893    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrtq2" event={"ID":"99c008bb-acfe-480a-a07c-9bccdce3d34b","Type":"ContainerStarted","Data":"3e9319ed953d5e492a920d175c821fd99352401b14aa5e9bdfb03ecf89349952"}
Jul 24 12:20:22.919903 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:22.919884    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54a7f8b4-055f-424b-87a5-d83333058107","Type":"ContainerStarted","Data":"253f2c8b0c37300ddb96b75deecf8d9bad3c1d1ff3d37f61f99492133c521190"}
Jul 24 12:20:22.920763 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:22.920745    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk" event={"ID":"789a6a57-4d93-47d3-abd7-201f0c2d53eb","Type":"ContainerStarted","Data":"628f5507c9af596480746bce601c7fe4b441d5cc11880e9ee52c0b960370ae63"}
Jul 24 12:20:22.921604 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:22.921587    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5" event={"ID":"215ec8d5-e824-4a7e-9b89-4b9ee4127682","Type":"ContainerStarted","Data":"eed3312888f7aab5446d825c947c0249b2130d7418c324b3a5010ae069c0e17e"}
Jul 24 12:20:22.923291 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:22.923273    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-86866dff5f-8ph8r"]
Jul 24 12:20:22.933112 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:22.933090    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:22.935753 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:22.935587    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 24 12:20:22.935753 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:22.935606    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-7fc5b\""
Jul 24 12:20:22.935753 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:22.935641    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 24 12:20:22.935753 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:22.935606    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 24 12:20:22.936032 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:22.935934    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 24 12:20:22.936089 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:22.936073    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-66l21jqiqssh7\""
Jul 24 12:20:22.938696 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:22.938677    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-86866dff5f-8ph8r"]
Jul 24 12:20:22.953918 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:22.953875    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55ff44d7df-rgr25" podStartSLOduration=2.953983094 podStartE2EDuration="8.953859216s" totalImagesPullingTime="5.999876122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:22.953328233 +0000 UTC m=+54.908333024" watchObservedRunningTime="2026-07-24 12:20:22.953859216 +0000 UTC m=+54.908864040"
Jul 24 12:20:23.007730 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.007461    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-audit-log\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.007730 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.007523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-secret-metrics-server-client-certs\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.007730 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.007550    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-metrics-server-audit-profiles\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.007730 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.007572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-client-ca-bundle\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.007730 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.007610    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-secret-metrics-server-tls\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.007730 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.007639    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.007730 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.007663    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhnq7\" (UniqueName: \"kubernetes.io/projected/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-kube-api-access-nhnq7\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.108457 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.108409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-secret-metrics-server-tls\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.108601 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.108474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.108601 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.108506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhnq7\" (UniqueName: \"kubernetes.io/projected/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-kube-api-access-nhnq7\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.108724 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.108676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-audit-log\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.108724 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.108706    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-secret-metrics-server-client-certs\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.108823 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.108729    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-metrics-server-audit-profiles\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.108823 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.108751    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-client-ca-bundle\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.109191 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.109158    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.109269 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.109191    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-audit-log\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.109917 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.109891    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-metrics-server-audit-profiles\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.111059 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.111039    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-secret-metrics-server-tls\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.111709 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.111692    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-secret-metrics-server-client-certs\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.111770 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.111733    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-client-ca-bundle\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.117027 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.117008    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhnq7\" (UniqueName: \"kubernetes.io/projected/a1c15d37-5a52-4f35-8f0a-fa629c497ad7-kube-api-access-nhnq7\") pod \"metrics-server-86866dff5f-8ph8r\" (UID: \"a1c15d37-5a52-4f35-8f0a-fa629c497ad7\") " pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.245604 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.245529    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:23.414872 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.414797    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-86866dff5f-8ph8r"]
Jul 24 12:20:23.423340 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.423310    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-jmsvs"]
Jul 24 12:20:23.428669 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:20:23.428633    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c15d37_5a52_4f35_8f0a_fa629c497ad7.slice/crio-b5be9d09f8d88a714eca38d9794fb037fcf634a3d9ce51ab1e0d103d487b9e2a WatchSource:0}: Error finding container b5be9d09f8d88a714eca38d9794fb037fcf634a3d9ce51ab1e0d103d487b9e2a: Status 404 returned error can't find the container with id b5be9d09f8d88a714eca38d9794fb037fcf634a3d9ce51ab1e0d103d487b9e2a
Jul 24 12:20:23.438670 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.438649    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmsvs"
Jul 24 12:20:23.441122 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.441095    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 24 12:20:23.444263 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.444205    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55ff44d7df-rgr25"]
Jul 24 12:20:23.444685 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.444636    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-jmsvs"]
Jul 24 12:20:23.444802 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.441977    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7lqv8\""
Jul 24 12:20:23.500663 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.500565    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57f58d6854-4j9wh"]
Jul 24 12:20:23.513797 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.513761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-jmsvs\" (UID: \"3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmsvs"
Jul 24 12:20:23.513967 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.513889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg7hr\" (UniqueName: \"kubernetes.io/projected/3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041-kube-api-access-pg7hr\") pod \"monitoring-plugin-68b7fc9574-jmsvs\" (UID: \"3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmsvs"
Jul 24 12:20:23.515037 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.515011    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.515898 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.515874    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57f58d6854-4j9wh"]
Jul 24 12:20:23.614962 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.614932    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-console-serving-cert\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.615340 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.614972    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh6vt\" (UniqueName: \"kubernetes.io/projected/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-kube-api-access-rh6vt\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.615340 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.615004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg7hr\" (UniqueName: \"kubernetes.io/projected/3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041-kube-api-access-pg7hr\") pod \"monitoring-plugin-68b7fc9574-jmsvs\" (UID: \"3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmsvs"
Jul 24 12:20:23.615340 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.615029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-console-config\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.615340 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.615064    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-console-oauth-config\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.615340 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.615108    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-service-ca\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.615340 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.615127    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-jmsvs\" (UID: \"3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmsvs"
Jul 24 12:20:23.615340 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.615165    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-trusted-ca-bundle\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.615340 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.615196    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-oauth-serving-cert\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.615736 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:23.615642    2581 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 24 12:20:23.615736 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:20:23.615702    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041-monitoring-plugin-cert podName:3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041 nodeName:}" failed. No retries permitted until 2026-07-24 12:20:24.11568312 +0000 UTC m=+56.070687867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-jmsvs" (UID: "3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041") : secret "monitoring-plugin-cert" not found
Jul 24 12:20:23.624773 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.624724    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg7hr\" (UniqueName: \"kubernetes.io/projected/3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041-kube-api-access-pg7hr\") pod \"monitoring-plugin-68b7fc9574-jmsvs\" (UID: \"3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmsvs"
Jul 24 12:20:23.716018 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.715982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-console-config\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.716200 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.716039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-console-oauth-config\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.716200 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.716099    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-service-ca\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.716200 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.716171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-trusted-ca-bundle\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.716341 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.716212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-oauth-serving-cert\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.716341 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.716253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-console-serving-cert\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.716341 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.716295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh6vt\" (UniqueName: \"kubernetes.io/projected/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-kube-api-access-rh6vt\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.717435 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.717398    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-trusted-ca-bundle\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.717559 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.717398    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-console-config\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.717620 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.717549    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-service-ca\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.717958 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.717937    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-oauth-serving-cert\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.720673 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.720589    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-console-oauth-config\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.720785 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.720748    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-console-serving-cert\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.730220 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.729197    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh6vt\" (UniqueName: \"kubernetes.io/projected/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-kube-api-access-rh6vt\") pod \"console-57f58d6854-4j9wh\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") " pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.827495 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.827458    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:23.872449 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.869861    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-77c9c7c646-cznc8"]
Jul 24 12:20:23.896876 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.896011    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-77c9c7c646-cznc8"]
Jul 24 12:20:23.896876 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.896134    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:23.900406 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.899652    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-5brvr\""
Jul 24 12:20:23.900406 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.899652    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 24 12:20:23.900406 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.899926    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 24 12:20:23.900406 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.900020    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 24 12:20:23.900406 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.900174    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 24 12:20:23.900406 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.900223    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 24 12:20:23.905602 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.905406    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 24 12:20:23.929015 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.928887    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5" event={"ID":"215ec8d5-e824-4a7e-9b89-4b9ee4127682","Type":"ContainerStarted","Data":"08917f995298c5774aedda576b6a5eb5c552c8d62891d271807851188c9fc3ae"}
Jul 24 12:20:23.929015 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.928963    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5" event={"ID":"215ec8d5-e824-4a7e-9b89-4b9ee4127682","Type":"ContainerStarted","Data":"f30a49094c0d49236bbc9d584851abdfbceff6805094529d6a5636eede44753a"}
Jul 24 12:20:23.930466 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.930437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r" event={"ID":"a1c15d37-5a52-4f35-8f0a-fa629c497ad7","Type":"ContainerStarted","Data":"b5be9d09f8d88a714eca38d9794fb037fcf634a3d9ce51ab1e0d103d487b9e2a"}
Jul 24 12:20:23.932359 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:23.932329    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rzcwk" event={"ID":"aa4a90db-e263-4e0d-ad83-9b0a60b282a0","Type":"ContainerStarted","Data":"a5f2b7f593d22f7c898a5fd7921b515c496472bdf11bd0e4238cfcebef4aefdb"}
Jul 24 12:20:24.020570 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.020524    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f736943-1b3c-493f-a2d0-f767b1794d2a-serving-certs-ca-bundle\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.020746 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.020720    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9f736943-1b3c-493f-a2d0-f767b1794d2a-secret-telemeter-client\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.020855 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.020807    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9f736943-1b3c-493f-a2d0-f767b1794d2a-telemeter-client-tls\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.020921 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.020903    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9f736943-1b3c-493f-a2d0-f767b1794d2a-federate-client-tls\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.020978 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.020949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f736943-1b3c-493f-a2d0-f767b1794d2a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.021557 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.021204    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt99k\" (UniqueName: \"kubernetes.io/projected/9f736943-1b3c-493f-a2d0-f767b1794d2a-kube-api-access-xt99k\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.021557 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.021291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f736943-1b3c-493f-a2d0-f767b1794d2a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.021557 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.021330    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f736943-1b3c-493f-a2d0-f767b1794d2a-metrics-client-ca\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.122331 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.122244    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f736943-1b3c-493f-a2d0-f767b1794d2a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.122331 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.122297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt99k\" (UniqueName: \"kubernetes.io/projected/9f736943-1b3c-493f-a2d0-f767b1794d2a-kube-api-access-xt99k\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.122585 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.122354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f736943-1b3c-493f-a2d0-f767b1794d2a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.122585 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.122386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f736943-1b3c-493f-a2d0-f767b1794d2a-metrics-client-ca\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.122585 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.122467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f736943-1b3c-493f-a2d0-f767b1794d2a-serving-certs-ca-bundle\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.122585 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.122537    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-jmsvs\" (UID: \"3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmsvs"
Jul 24 12:20:24.122794 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.122587    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9f736943-1b3c-493f-a2d0-f767b1794d2a-secret-telemeter-client\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.122794 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.122652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9f736943-1b3c-493f-a2d0-f767b1794d2a-telemeter-client-tls\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.122794 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.122688    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9f736943-1b3c-493f-a2d0-f767b1794d2a-federate-client-tls\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.123918 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.123891    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f736943-1b3c-493f-a2d0-f767b1794d2a-metrics-client-ca\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.125938 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.124908    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f736943-1b3c-493f-a2d0-f767b1794d2a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.126909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.126863    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f736943-1b3c-493f-a2d0-f767b1794d2a-serving-certs-ca-bundle\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.129337 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.129278    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9f736943-1b3c-493f-a2d0-f767b1794d2a-secret-telemeter-client\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.129460 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.129444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9f736943-1b3c-493f-a2d0-f767b1794d2a-federate-client-tls\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.130901 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.130875    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-jmsvs\" (UID: \"3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmsvs"
Jul 24 12:20:24.131866 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.131840    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f736943-1b3c-493f-a2d0-f767b1794d2a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.132320 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.132299    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9f736943-1b3c-493f-a2d0-f767b1794d2a-telemeter-client-tls\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.134975 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.134952    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt99k\" (UniqueName: \"kubernetes.io/projected/9f736943-1b3c-493f-a2d0-f767b1794d2a-kube-api-access-xt99k\") pod \"telemeter-client-77c9c7c646-cznc8\" (UID: \"9f736943-1b3c-493f-a2d0-f767b1794d2a\") " pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.211526 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.211487    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8"
Jul 24 12:20:24.352821 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.352783    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmsvs"
Jul 24 12:20:24.845693 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.845646    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 12:20:24.863165 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.862656    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:24.866245 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.866219    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 12:20:24.867686 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.867663    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9f1jekgljt1t5\""
Jul 24 12:20:24.867816 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.867770    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 24 12:20:24.868475 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.867893    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 24 12:20:24.868475 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.867970    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 24 12:20:24.868475 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.868089    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 24 12:20:24.868475 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.868150    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 24 12:20:24.868475 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.867974    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2ct2z\""
Jul 24 12:20:24.868475 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.868211    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 24 12:20:24.868475 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.868282    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 24 12:20:24.868475 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.867890    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 24 12:20:24.868475 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.868333    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 24 12:20:24.868475 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.868404    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 24 12:20:24.870805 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.870604    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 24 12:20:24.871505 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.871108    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 24 12:20:24.873295 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.873274    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 24 12:20:24.877374 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.877351    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 24 12:20:24.929765 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.929728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:24.929765 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.929769    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:24.929979 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.929808    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:24.929979 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.929866    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-web-config\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:24.929979 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.929909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-config-out\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:24.929979 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.929938    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:24.929979 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.929967    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:24.930216 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.930063    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:24.930216 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.930089    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:24.930216 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.930118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msglz\" (UniqueName: \"kubernetes.io/projected/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-kube-api-access-msglz\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:24.930216 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.930186    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:24.930216 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.930213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:24.930503 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.930232    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:24.930503 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.930263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:24.930503 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.930300    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-config\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:24.930503 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.930315    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:24.930503 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.930333    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:24.930503 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.930369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:24.930503 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.930386    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:24.930503 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:24.930409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.031970 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.031929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.031970 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.031981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.032249 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.032014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msglz\" (UniqueName: \"kubernetes.io/projected/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-kube-api-access-msglz\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.032249 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.032066    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.032249 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.032100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.032249 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.032123    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.032249 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.032165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.032249 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.032194    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-config\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.032249 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.032218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.032249 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.032247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.035336 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.032281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.035336 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.032302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.035336 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.032321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.035336 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.032355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.035336 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.032374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.035336 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.032400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.035336 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.032441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-web-config\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.035336 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.032458    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-config-out\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.035336 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.032476    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.035336 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.032505    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.035336 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.033036    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.035336 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.033750    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.037437 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.037009    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.037437 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.037195    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.037437 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.037369    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.038004 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.037981    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.040043 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.040017    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.040546 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.040199    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.040546 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.040355    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-config-out\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.040916 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.040878    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.041214 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.041193    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.043929 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.043887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.044257 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.044221    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.045566 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.045322    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.045566 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.045512    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msglz\" (UniqueName: \"kubernetes.io/projected/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-kube-api-access-msglz\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.046841 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.046816    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-config\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.052698 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.052667    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.052808 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.052704    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-web-config\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.052887 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.052866    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.054911 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.054886    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/997cf464-b4ac-4dca-a03b-2c5026a6c2b2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"997cf464-b4ac-4dca-a03b-2c5026a6c2b2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.182593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.182260    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:25.207410 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:25.207380    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:26.467721 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:26.467689    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-jmsvs"]
Jul 24 12:20:26.896668 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:26.896644    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8dqxn"
Jul 24 12:20:28.820067 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:28.820029    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzz5k"
Jul 24 12:20:28.949704 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:20:28.949665    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab6ce6c_61a7_49a6_8e86_fc76f9c2e041.slice/crio-12e76962a4e82ca03668270c7d547acdab1f87f4ca1dccb8a78e1f7209bed26d WatchSource:0}: Error finding container 12e76962a4e82ca03668270c7d547acdab1f87f4ca1dccb8a78e1f7209bed26d: Status 404 returned error can't find the container with id 12e76962a4e82ca03668270c7d547acdab1f87f4ca1dccb8a78e1f7209bed26d
Jul 24 12:20:28.955851 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:28.955819    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmsvs" event={"ID":"3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041","Type":"ContainerStarted","Data":"12e76962a4e82ca03668270c7d547acdab1f87f4ca1dccb8a78e1f7209bed26d"}
Jul 24 12:20:29.111612 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.111582    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-77c9c7c646-cznc8"]
Jul 24 12:20:29.129694 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:20:29.129666    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f736943_1b3c_493f_a2d0_f767b1794d2a.slice/crio-42cffbb8e6ec14c2e58d57b64ed157411a2f159caa78c51ffbdb7b2f7efe5fd5 WatchSource:0}: Error finding container 42cffbb8e6ec14c2e58d57b64ed157411a2f159caa78c51ffbdb7b2f7efe5fd5: Status 404 returned error can't find the container with id 42cffbb8e6ec14c2e58d57b64ed157411a2f159caa78c51ffbdb7b2f7efe5fd5
Jul 24 12:20:29.136889 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.135832    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57f58d6854-4j9wh"]
Jul 24 12:20:29.151742 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:20:29.151382    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb250d142_2d1e_40d5_bf3a_73fa89ae42dc.slice/crio-ee7c4acda11fe3e8051112ec19bb49238bed5d0a3e81e10a381342503b2d1b52 WatchSource:0}: Error finding container ee7c4acda11fe3e8051112ec19bb49238bed5d0a3e81e10a381342503b2d1b52: Status 404 returned error can't find the container with id ee7c4acda11fe3e8051112ec19bb49238bed5d0a3e81e10a381342503b2d1b52
Jul 24 12:20:29.204222 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.201673    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 12:20:29.257719 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:20:29.257201    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997cf464_b4ac_4dca_a03b_2c5026a6c2b2.slice/crio-624203f9e93c81d65ae2244f5307ec1ca3bc25029cba0f527db7b99f68f83765 WatchSource:0}: Error finding container 624203f9e93c81d65ae2244f5307ec1ca3bc25029cba0f527db7b99f68f83765: Status 404 returned error can't find the container with id 624203f9e93c81d65ae2244f5307ec1ca3bc25029cba0f527db7b99f68f83765
Jul 24 12:20:29.964350 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.964314    2581 generic.go:356] "Generic (PLEG): container finished" podID="54a7f8b4-055f-424b-87a5-d83333058107" containerID="2aa2620e3f7436c64e07f1cd42182934927a1991ca94f0faad9e5c4db757b833" exitCode=0
Jul 24 12:20:29.964865 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.964426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54a7f8b4-055f-424b-87a5-d83333058107","Type":"ContainerDied","Data":"2aa2620e3f7436c64e07f1cd42182934927a1991ca94f0faad9e5c4db757b833"}
Jul 24 12:20:29.969639 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.968157    2581 generic.go:356] "Generic (PLEG): container finished" podID="997cf464-b4ac-4dca-a03b-2c5026a6c2b2" containerID="d6b191b452e29287e6e068cef37278334a11c7f9f7ef2f4aa23beab3195a3673" exitCode=0
Jul 24 12:20:29.969639 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.968377    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"997cf464-b4ac-4dca-a03b-2c5026a6c2b2","Type":"ContainerDied","Data":"d6b191b452e29287e6e068cef37278334a11c7f9f7ef2f4aa23beab3195a3673"}
Jul 24 12:20:29.969639 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.968468    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"997cf464-b4ac-4dca-a03b-2c5026a6c2b2","Type":"ContainerStarted","Data":"624203f9e93c81d65ae2244f5307ec1ca3bc25029cba0f527db7b99f68f83765"}
Jul 24 12:20:29.970874 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.970683    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8" event={"ID":"9f736943-1b3c-493f-a2d0-f767b1794d2a","Type":"ContainerStarted","Data":"42cffbb8e6ec14c2e58d57b64ed157411a2f159caa78c51ffbdb7b2f7efe5fd5"}
Jul 24 12:20:29.973495 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.973473    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk" event={"ID":"789a6a57-4d93-47d3-abd7-201f0c2d53eb","Type":"ContainerStarted","Data":"4efaacf46cce01bd1df72c697ec53b78ff13b2fd82f1e864337c41dd4091329d"}
Jul 24 12:20:29.973675 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.973579    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk" event={"ID":"789a6a57-4d93-47d3-abd7-201f0c2d53eb","Type":"ContainerStarted","Data":"d1fddfeb27742a848632cb21477e624becf3de7ec433e2f06c7459da0c5a41e2"}
Jul 24 12:20:29.973675 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.973631    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk" event={"ID":"789a6a57-4d93-47d3-abd7-201f0c2d53eb","Type":"ContainerStarted","Data":"679c9f0e71948d2cb5eb4368e096937992da0df82174a53acc15a8077d6dfbc2"}
Jul 24 12:20:29.977747 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.977241    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5" event={"ID":"215ec8d5-e824-4a7e-9b89-4b9ee4127682","Type":"ContainerStarted","Data":"116b0092bb25e5cab4f80a68a0e208101e317c80afb2cb17b04e48ac2692fd23"}
Jul 24 12:20:29.981043 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.981018    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr" event={"ID":"18cdccc7-5d25-4e5b-af0e-b0a40d851053","Type":"ContainerStarted","Data":"16fadf9391329aa4844a46cf02202bed28af12f875b04262cca0f016a3acd3be"}
Jul 24 12:20:29.981124 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.981064    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr" event={"ID":"18cdccc7-5d25-4e5b-af0e-b0a40d851053","Type":"ContainerStarted","Data":"6e9be0d4ab568013e9f29dde2becff5d6720107f9797f9ec1552d02e2652cbcc"}
Jul 24 12:20:29.981124 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.981087    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr" event={"ID":"18cdccc7-5d25-4e5b-af0e-b0a40d851053","Type":"ContainerStarted","Data":"5f7e4f49be7263e19425bf6690161b62d24c70f5b747f2133b81458ca9384c19"}
Jul 24 12:20:29.982958 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.982934    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r" event={"ID":"a1c15d37-5a52-4f35-8f0a-fa629c497ad7","Type":"ContainerStarted","Data":"39a539551242d2f00646aef97db2f93b16c362b89193dbffa87add219cdf3ec7"}
Jul 24 12:20:29.985737 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.985710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rzcwk" event={"ID":"aa4a90db-e263-4e0d-ad83-9b0a60b282a0","Type":"ContainerStarted","Data":"4d0034e8db489ae51c0fbcebabb8b3630fdabc381a348a22d416652ddd3cc568"}
Jul 24 12:20:29.988163 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.988075    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f58d6854-4j9wh" event={"ID":"b250d142-2d1e-40d5-bf3a-73fa89ae42dc","Type":"ContainerStarted","Data":"996067b6e4e7ef8cb70b69ae0cba6f0f38fc14df4addbbb52c868fed89ab9ae0"}
Jul 24 12:20:29.988163 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.988114    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f58d6854-4j9wh" event={"ID":"b250d142-2d1e-40d5-bf3a-73fa89ae42dc","Type":"ContainerStarted","Data":"ee7c4acda11fe3e8051112ec19bb49238bed5d0a3e81e10a381342503b2d1b52"}
Jul 24 12:20:29.990709 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.990633    2581 generic.go:356] "Generic (PLEG): container finished" podID="99c008bb-acfe-480a-a07c-9bccdce3d34b" containerID="4c6dea1e33259a3e15c874d50cd081a54d0d8a7d0287c87a40b2cc8b07b59c8d" exitCode=0
Jul 24 12:20:29.990709 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:29.990670    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrtq2" event={"ID":"99c008bb-acfe-480a-a07c-9bccdce3d34b","Type":"ContainerDied","Data":"4c6dea1e33259a3e15c874d50cd081a54d0d8a7d0287c87a40b2cc8b07b59c8d"}
Jul 24 12:20:30.003889 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:30.003818    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r" podStartSLOduration=2.4087421190000002 podStartE2EDuration="8.003793746s" totalImagesPullingTime="5.595051627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:30.003005601 +0000 UTC m=+61.958010367" watchObservedRunningTime="2026-07-24 12:20:30.003793746 +0000 UTC m=+61.958798501"
Jul 24 12:20:30.020166 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:30.020107    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cbkfr" podStartSLOduration=5.719932372 podStartE2EDuration="12.020085818s" totalImagesPullingTime="6.300153446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:30.019091865 +0000 UTC m=+61.974096629" watchObservedRunningTime="2026-07-24 12:20:30.020085818 +0000 UTC m=+61.975090574"
Jul 24 12:20:30.035537 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:30.035475    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-q22x5" podStartSLOduration=6.175429745 podStartE2EDuration="12.035454176s" totalImagesPullingTime="5.860024431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:30.033235664 +0000 UTC m=+61.988240429" watchObservedRunningTime="2026-07-24 12:20:30.035454176 +0000 UTC m=+61.990458941"
Jul 24 12:20:30.051337 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:30.051285    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rzcwk" podStartSLOduration=2.28952158 podStartE2EDuration="22.051270415s" totalImagesPullingTime="19.761748835s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:20:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:30.049630143 +0000 UTC m=+62.004634919" watchObservedRunningTime="2026-07-24 12:20:30.051270415 +0000 UTC m=+62.006275179"
Jul 24 12:20:30.101260 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:30.101210    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57f58d6854-4j9wh" podStartSLOduration=7.101194473 podStartE2EDuration="7.101194473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:30.100016911 +0000 UTC m=+62.055021689" watchObservedRunningTime="2026-07-24 12:20:30.101194473 +0000 UTC m=+62.056199235"
Jul 24 12:20:31.002653 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:31.002566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrtq2" event={"ID":"99c008bb-acfe-480a-a07c-9bccdce3d34b","Type":"ContainerStarted","Data":"46d429246f2a550290c4ff22e571b2b9e4b1fe1b892384946b566301462d9417"}
Jul 24 12:20:31.002653 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:31.002624    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rrtq2" event={"ID":"99c008bb-acfe-480a-a07c-9bccdce3d34b","Type":"ContainerStarted","Data":"1f15b339d28915d9d5e2d02cc157860762c19413b023427e2688a0a3afb6ff5d"}
Jul 24 12:20:31.022943 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:31.022878    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rrtq2" podStartSLOduration=6.376229711 podStartE2EDuration="13.022860248s" totalImagesPullingTime="6.476009865s" totalInitContainerRuntime="170.620672ms" isStatefulPod=true podCreationTimestamp="2026-07-24 12:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:31.020576699 +0000 UTC m=+62.975581488" watchObservedRunningTime="2026-07-24 12:20:31.022860248 +0000 UTC m=+62.977865010"
Jul 24 12:20:32.024759 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:32.024723    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk" event={"ID":"789a6a57-4d93-47d3-abd7-201f0c2d53eb","Type":"ContainerStarted","Data":"76dc834b3239ccd4849a6e6a12121ae4c91cc51236759e05ed5f596cb014253a"}
Jul 24 12:20:32.025346 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:32.025320    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk" event={"ID":"789a6a57-4d93-47d3-abd7-201f0c2d53eb","Type":"ContainerStarted","Data":"7ad9a8271fe11e3cc977f34f430d82a2857f71aa6b089c8c2572b3edea549c83"}
Jul 24 12:20:32.028476 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:32.028443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54a7f8b4-055f-424b-87a5-d83333058107","Type":"ContainerStarted","Data":"80d0c5a4b3d59d114096849f404adf964b554cc38d4a6662eea8d0f3c2971faf"}
Jul 24 12:20:32.028550 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:32.028524    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54a7f8b4-055f-424b-87a5-d83333058107","Type":"ContainerStarted","Data":"0add7cd2fd8308526947b5864a7fdb4ad84cda958b845129a5dc82142e236621"}
Jul 24 12:20:32.031894 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:32.031686    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmsvs" event={"ID":"3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041","Type":"ContainerStarted","Data":"29d3942633aed9061a3efc10957e3b0e3ae6304f9c71e41ff2d58c920ff58e04"}
Jul 24 12:20:32.032245 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:32.032215    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmsvs"
Jul 24 12:20:32.036929 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:32.036900    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8" event={"ID":"9f736943-1b3c-493f-a2d0-f767b1794d2a","Type":"ContainerStarted","Data":"9d7f74a2975210d0341aee95b3a73c46702ff8549cd6508f087960895d75717e"}
Jul 24 12:20:32.037036 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:32.036955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8" event={"ID":"9f736943-1b3c-493f-a2d0-f767b1794d2a","Type":"ContainerStarted","Data":"9ff5e886f71d49902aa40849167bd4d6f338851efee288aadd3e6d082f741189"}
Jul 24 12:20:32.040679 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:32.040579    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmsvs"
Jul 24 12:20:32.048910 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:32.048850    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jmsvs" podStartSLOduration=7.443235506 podStartE2EDuration="9.048831718s" totalImagesPullingTime="1.605596212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:32.047925992 +0000 UTC m=+64.002930757" watchObservedRunningTime="2026-07-24 12:20:32.048831718 +0000 UTC m=+64.003836462"
Jul 24 12:20:33.048959 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:33.048914    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk" event={"ID":"789a6a57-4d93-47d3-abd7-201f0c2d53eb","Type":"ContainerStarted","Data":"6ba9d00c6a166c6887cf8a0ddf2fe374b3fa606c44e2e06495e5358baf5af0ff"}
Jul 24 12:20:33.049444 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:33.049197    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:33.054072 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:33.053208    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54a7f8b4-055f-424b-87a5-d83333058107","Type":"ContainerStarted","Data":"187ed1ff67077c45adef0c81bd0637d3df6724fb301a0be8e872b6cfb12217f5"}
Jul 24 12:20:33.054072 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:33.053282    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54a7f8b4-055f-424b-87a5-d83333058107","Type":"ContainerStarted","Data":"877c385995bf7d5ceb48f2c892cf7c11f91da9722ffc65eff7053b98cb85f3c1"}
Jul 24 12:20:33.054072 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:33.053327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54a7f8b4-055f-424b-87a5-d83333058107","Type":"ContainerStarted","Data":"2b1ade0032dba43037992ec0e27745b203a948b021637c1107979cbe06f181ac"}
Jul 24 12:20:33.054072 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:33.053369    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54a7f8b4-055f-424b-87a5-d83333058107","Type":"ContainerStarted","Data":"a9275b32573de7141296228596accac6b8198ab0553b0633c03a6643582e6e8d"}
Jul 24 12:20:33.055690 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:33.055661    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8" event={"ID":"9f736943-1b3c-493f-a2d0-f767b1794d2a","Type":"ContainerStarted","Data":"c81f9fa043cc041239f589b58e8d3b974fae577f7a8ad2ce5a9e9c6161f96c61"}
Jul 24 12:20:33.075609 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:33.075541    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk" podStartSLOduration=5.610868903 podStartE2EDuration="13.07552079s" totalImagesPullingTime="7.464651887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:20:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:33.073925151 +0000 UTC m=+65.028929915" watchObservedRunningTime="2026-07-24 12:20:33.07552079 +0000 UTC m=+65.030525555"
Jul 24 12:20:33.100892 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:33.100841    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-77c9c7c646-cznc8" podStartSLOduration=7.461961306 podStartE2EDuration="10.100825103s" totalImagesPullingTime="2.638863797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:33.099108683 +0000 UTC m=+65.054113664" watchObservedRunningTime="2026-07-24 12:20:33.100825103 +0000 UTC m=+65.055829873"
Jul 24 12:20:33.127457 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:33.125019    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=6.03368594 podStartE2EDuration="14.124980904s" totalImagesPullingTime="8.072495419s" totalInitContainerRuntime="18.799545ms" isStatefulPod=false podCreationTimestamp="2026-07-24 12:20:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:33.124383425 +0000 UTC m=+65.079388221" watchObservedRunningTime="2026-07-24 12:20:33.124980904 +0000 UTC m=+65.079985667"
Jul 24 12:20:33.828371 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:33.828338    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:33.828541 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:33.828445    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:33.835756 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:33.835735    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:34.038767 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.038739    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57f58d6854-4j9wh"]
Jul 24 12:20:34.071780 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.070072    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"997cf464-b4ac-4dca-a03b-2c5026a6c2b2","Type":"ContainerStarted","Data":"d48182e5361d2b0c4a561ef03f6d24fdf6956fe3854680058819703ec64e4c5c"}
Jul 24 12:20:34.071780 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.070154    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"997cf464-b4ac-4dca-a03b-2c5026a6c2b2","Type":"ContainerStarted","Data":"50bbd062fa2ac4f6869f8b8f2e77d6488275fff847e954b94409c681ffe5fe43"}
Jul 24 12:20:34.071780 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.070197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"997cf464-b4ac-4dca-a03b-2c5026a6c2b2","Type":"ContainerStarted","Data":"5d06fd3fa4835dcfdebb930f130fa37a9a033183b58d7481c1fa4b4babbf01d1"}
Jul 24 12:20:34.071780 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.070235    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"997cf464-b4ac-4dca-a03b-2c5026a6c2b2","Type":"ContainerStarted","Data":"7826140ac17ce89832cbbd3b069f4ee1bfce64a0c5d4c78d3556ffaad0e07965"}
Jul 24 12:20:34.078091 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.078029    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:20:34.084496 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.082692    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bf94dd8db-zgnxm"]
Jul 24 12:20:34.087274 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.087247    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-85c5b8bcfb-bkqjk"
Jul 24 12:20:34.087493 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.087477    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.100000 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.098489    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bf94dd8db-zgnxm"]
Jul 24 12:20:34.238718 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.238626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs\") pod \"network-metrics-daemon-wtb2z\" (UID: \"bda12f0d-5325-4258-a601-381a0751907a\") " pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:20:34.238887 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.238745    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-trusted-ca-bundle\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.238887 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.238845    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7651e1da-a87d-4427-982d-b588b77dddab-console-oauth-config\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.239591 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.239292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-console-config\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.239591 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.239348    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2w9r\" (UniqueName: \"kubernetes.io/projected/7651e1da-a87d-4427-982d-b588b77dddab-kube-api-access-b2w9r\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.239819 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.239655    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7651e1da-a87d-4427-982d-b588b77dddab-console-serving-cert\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.239819 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.239735    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-service-ca\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.239983 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.239880    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-oauth-serving-cert\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.240729 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.240706    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 12:20:34.252222 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.252195    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bda12f0d-5325-4258-a601-381a0751907a-metrics-certs\") pod \"network-metrics-daemon-wtb2z\" (UID: \"bda12f0d-5325-4258-a601-381a0751907a\") " pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:20:34.341620 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.341583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-service-ca\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.341797 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.341637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm2k2\" (UniqueName: \"kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2\") pod \"network-check-target-8294s\" (UID: \"edc2499b-10da-416b-831d-52f00cbd026e\") " pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:20:34.341797 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.341657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-oauth-serving-cert\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.341797 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.341706    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret\") pod \"global-pull-secret-syncer-qrzvj\" (UID: \"d98f89d0-11ef-4c6b-b973-64267283f1da\") " pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:20:34.341797 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.341742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-trusted-ca-bundle\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.341797 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.341772    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7651e1da-a87d-4427-982d-b588b77dddab-console-oauth-config\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.342076 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.341824    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-console-config\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.342076 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.341846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2w9r\" (UniqueName: \"kubernetes.io/projected/7651e1da-a87d-4427-982d-b588b77dddab-kube-api-access-b2w9r\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.342076 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.341888    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7651e1da-a87d-4427-982d-b588b77dddab-console-serving-cert\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.342550 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.342521    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-service-ca\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.342652 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.342528    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-oauth-serving-cert\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.342652 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.342597    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-console-config\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.342984 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.342961    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-trusted-ca-bundle\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.343616 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.343595    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 12:20:34.343711 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.343661    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 12:20:34.345041 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.344998    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7651e1da-a87d-4427-982d-b588b77dddab-console-serving-cert\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.346341 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.346322    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7651e1da-a87d-4427-982d-b588b77dddab-console-oauth-config\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.351731 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.351713    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2w9r\" (UniqueName: \"kubernetes.io/projected/7651e1da-a87d-4427-982d-b588b77dddab-kube-api-access-b2w9r\") pod \"console-6bf94dd8db-zgnxm\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") " pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.354026 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.354011    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 12:20:34.354666 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.354649    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d98f89d0-11ef-4c6b-b973-64267283f1da-original-pull-secret\") pod \"global-pull-secret-syncer-qrzvj\" (UID: \"d98f89d0-11ef-4c6b-b973-64267283f1da\") " pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:20:34.366318 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.366283    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm2k2\" (UniqueName: \"kubernetes.io/projected/edc2499b-10da-416b-831d-52f00cbd026e-kube-api-access-bm2k2\") pod \"network-check-target-8294s\" (UID: \"edc2499b-10da-416b-831d-52f00cbd026e\") " pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:20:34.409432 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.409375    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:34.454594 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.454565    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k4xp5\""
Jul 24 12:20:34.458231 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.458207    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zhxrp\""
Jul 24 12:20:34.460991 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.460724    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrzvj"
Jul 24 12:20:34.462887 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.462811    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:20:34.466662 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.466575    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wtb2z"
Jul 24 12:20:34.576797 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.576717    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bf94dd8db-zgnxm"]
Jul 24 12:20:34.582618 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:20:34.582537    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7651e1da_a87d_4427_982d_b588b77dddab.slice/crio-784591f75ec7ff56aca10f8d78d8651252035501abe20e543edead01fcd46d46 WatchSource:0}: Error finding container 784591f75ec7ff56aca10f8d78d8651252035501abe20e543edead01fcd46d46: Status 404 returned error can't find the container with id 784591f75ec7ff56aca10f8d78d8651252035501abe20e543edead01fcd46d46
Jul 24 12:20:34.670965 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.668956    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qrzvj"]
Jul 24 12:20:34.674064 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:20:34.673741    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98f89d0_11ef_4c6b_b973_64267283f1da.slice/crio-4fec991de6ffd3fc59a661344b97b2e1f91a7d67102bb9cd99426ced69fecbb5 WatchSource:0}: Error finding container 4fec991de6ffd3fc59a661344b97b2e1f91a7d67102bb9cd99426ced69fecbb5: Status 404 returned error can't find the container with id 4fec991de6ffd3fc59a661344b97b2e1f91a7d67102bb9cd99426ced69fecbb5
Jul 24 12:20:34.682965 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.682881    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wtb2z"]
Jul 24 12:20:34.714817 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:34.714732    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8294s"]
Jul 24 12:20:34.717595 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:20:34.717561    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc2499b_10da_416b_831d_52f00cbd026e.slice/crio-02450de0e050b475a914f21e03081a3ac14e13f9578f009ba474e4b9e5fad42b WatchSource:0}: Error finding container 02450de0e050b475a914f21e03081a3ac14e13f9578f009ba474e4b9e5fad42b: Status 404 returned error can't find the container with id 02450de0e050b475a914f21e03081a3ac14e13f9578f009ba474e4b9e5fad42b
Jul 24 12:20:35.075861 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:35.075825    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qrzvj" event={"ID":"d98f89d0-11ef-4c6b-b973-64267283f1da","Type":"ContainerStarted","Data":"4fec991de6ffd3fc59a661344b97b2e1f91a7d67102bb9cd99426ced69fecbb5"}
Jul 24 12:20:35.080106 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:35.080068    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"997cf464-b4ac-4dca-a03b-2c5026a6c2b2","Type":"ContainerStarted","Data":"18de44a5d588d0a7bd48eb2ec14ffe8a068fdb8733a14a6a2a6e92f6a7949f01"}
Jul 24 12:20:35.080253 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:35.080152    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"997cf464-b4ac-4dca-a03b-2c5026a6c2b2","Type":"ContainerStarted","Data":"24cbc4d86658d0820392e74b0372e1a6812536e64db5b733707739368264f9c4"}
Jul 24 12:20:35.081434 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:35.081390    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8294s" event={"ID":"edc2499b-10da-416b-831d-52f00cbd026e","Type":"ContainerStarted","Data":"02450de0e050b475a914f21e03081a3ac14e13f9578f009ba474e4b9e5fad42b"}
Jul 24 12:20:35.082934 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:35.082905    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bf94dd8db-zgnxm" event={"ID":"7651e1da-a87d-4427-982d-b588b77dddab","Type":"ContainerStarted","Data":"feb5a0ff65b830195e490721872367dd62a00c225d8a4452a31565244470f047"}
Jul 24 12:20:35.083043 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:35.082956    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bf94dd8db-zgnxm" event={"ID":"7651e1da-a87d-4427-982d-b588b77dddab","Type":"ContainerStarted","Data":"784591f75ec7ff56aca10f8d78d8651252035501abe20e543edead01fcd46d46"}
Jul 24 12:20:35.084076 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:35.084055    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wtb2z" event={"ID":"bda12f0d-5325-4258-a601-381a0751907a","Type":"ContainerStarted","Data":"bb4e6f22d44ba952d29af090b20dda890dfeb51cac43a114ebaeadf659b676b7"}
Jul 24 12:20:35.112925 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:35.112859    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=7.327515355 podStartE2EDuration="11.112835878s" totalImagesPullingTime="3.759869564s" totalInitContainerRuntime="25.450959ms" isStatefulPod=false podCreationTimestamp="2026-07-24 12:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:35.110090069 +0000 UTC m=+67.065094831" watchObservedRunningTime="2026-07-24 12:20:35.112835878 +0000 UTC m=+67.067840642"
Jul 24 12:20:35.129039 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:35.128976    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bf94dd8db-zgnxm" podStartSLOduration=1.128954015 podStartE2EDuration="1.128954015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:35.127189735 +0000 UTC m=+67.082194513" watchObservedRunningTime="2026-07-24 12:20:35.128954015 +0000 UTC m=+67.083958777"
Jul 24 12:20:35.183509 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:35.183481    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:20:36.091131 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:36.091047    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wtb2z" event={"ID":"bda12f0d-5325-4258-a601-381a0751907a","Type":"ContainerStarted","Data":"2f954858e677cc093dc9c4dfa0301c625500044594faabd149152e722186aae5"}
Jul 24 12:20:36.091131 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:36.091122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wtb2z" event={"ID":"bda12f0d-5325-4258-a601-381a0751907a","Type":"ContainerStarted","Data":"46cf2a63e0d2210d692f99e3396d3a7c1ffe63a39f0a97e59bb21cd9f20cef03"}
Jul 24 12:20:36.105508 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:36.105403    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wtb2z" podStartSLOduration=67.001188271 podStartE2EDuration="1m8.105380012s" totalImagesPullingTime="1.104191741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:36.104640703 +0000 UTC m=+68.059645477" watchObservedRunningTime="2026-07-24 12:20:36.105380012 +0000 UTC m=+68.060384769"
Jul 24 12:20:39.108625 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:39.108580    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8294s" event={"ID":"edc2499b-10da-416b-831d-52f00cbd026e","Type":"ContainerStarted","Data":"14dcd4ff5c5d9c39e5841ddfc06bc0450c61d06e41a826907542eadf1585a5b9"}
Jul 24 12:20:39.109081 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:39.108698    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:20:39.130554 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:39.130496    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8294s" podStartSLOduration=67.351218179 podStartE2EDuration="1m11.130476964s" totalImagesPullingTime="3.779258785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:39.129251987 +0000 UTC m=+71.084256767" watchObservedRunningTime="2026-07-24 12:20:39.130476964 +0000 UTC m=+71.085481727"
Jul 24 12:20:41.119649 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:41.119606    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qrzvj" event={"ID":"d98f89d0-11ef-4c6b-b973-64267283f1da","Type":"ContainerStarted","Data":"66af137c16b27fe08bb08d193f88cd1c066bac9d602e86823b302d82fe5d9ca7"}
Jul 24 12:20:41.134887 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:41.134832    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qrzvj" podStartSLOduration=67.153399401 podStartE2EDuration="1m13.134812329s" totalImagesPullingTime="5.981412928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 12:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:20:41.134111801 +0000 UTC m=+73.089116561" watchObservedRunningTime="2026-07-24 12:20:41.134812329 +0000 UTC m=+73.089817092"
Jul 24 12:20:43.246017 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:43.245973    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:43.246552 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:43.246055    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:20:44.409826 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:44.409790    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:44.410204 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:44.409859    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:44.414330 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:44.414309    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:45.140211 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:45.140183    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:20:49.967197 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:49.967026    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55ff44d7df-rgr25" podUID="5750adc2-c7a4-466b-a73a-662a71446c4a" containerName="console" containerID="cri-o://342fdcebc5ffb1c9a6b224d48895a6128f534208e7c8c9dfcb61974f9aeccd97" gracePeriod=15
Jul 24 12:20:50.158240 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.158208    2581 generic.go:356] "Generic (PLEG): container finished" podID="5750adc2-c7a4-466b-a73a-662a71446c4a" containerID="342fdcebc5ffb1c9a6b224d48895a6128f534208e7c8c9dfcb61974f9aeccd97" exitCode=0
Jul 24 12:20:50.158394 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.158260    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55ff44d7df-rgr25" event={"ID":"5750adc2-c7a4-466b-a73a-662a71446c4a","Type":"ContainerDied","Data":"342fdcebc5ffb1c9a6b224d48895a6128f534208e7c8c9dfcb61974f9aeccd97"}
Jul 24 12:20:50.747367 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.747330    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:50.802232 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.802207    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-service-ca\") pod \"5750adc2-c7a4-466b-a73a-662a71446c4a\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") "
Jul 24 12:20:50.802232 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.802237    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5750adc2-c7a4-466b-a73a-662a71446c4a-console-oauth-config\") pod \"5750adc2-c7a4-466b-a73a-662a71446c4a\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") "
Jul 24 12:20:50.802470 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.802308    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-trusted-ca-bundle\") pod \"5750adc2-c7a4-466b-a73a-662a71446c4a\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") "
Jul 24 12:20:50.802470 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.802326    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rktl\" (UniqueName: \"kubernetes.io/projected/5750adc2-c7a4-466b-a73a-662a71446c4a-kube-api-access-4rktl\") pod \"5750adc2-c7a4-466b-a73a-662a71446c4a\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") "
Jul 24 12:20:50.802470 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.802366    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-oauth-serving-cert\") pod \"5750adc2-c7a4-466b-a73a-662a71446c4a\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") "
Jul 24 12:20:50.802470 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.802410    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-console-config\") pod \"5750adc2-c7a4-466b-a73a-662a71446c4a\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") "
Jul 24 12:20:50.802669 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.802505    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5750adc2-c7a4-466b-a73a-662a71446c4a-console-serving-cert\") pod \"5750adc2-c7a4-466b-a73a-662a71446c4a\" (UID: \"5750adc2-c7a4-466b-a73a-662a71446c4a\") "
Jul 24 12:20:50.802736 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.802697    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-service-ca" (OuterVolumeSpecName: "service-ca") pod "5750adc2-c7a4-466b-a73a-662a71446c4a" (UID: "5750adc2-c7a4-466b-a73a-662a71446c4a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:20:50.802848 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.802826    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5750adc2-c7a4-466b-a73a-662a71446c4a" (UID: "5750adc2-c7a4-466b-a73a-662a71446c4a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:20:50.802907 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.802836    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-service-ca\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:20:50.802907 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.802853    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-console-config" (OuterVolumeSpecName: "console-config") pod "5750adc2-c7a4-466b-a73a-662a71446c4a" (UID: "5750adc2-c7a4-466b-a73a-662a71446c4a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:20:50.802907 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.802861    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5750adc2-c7a4-466b-a73a-662a71446c4a" (UID: "5750adc2-c7a4-466b-a73a-662a71446c4a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:20:50.804883 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.804854    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5750adc2-c7a4-466b-a73a-662a71446c4a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5750adc2-c7a4-466b-a73a-662a71446c4a" (UID: "5750adc2-c7a4-466b-a73a-662a71446c4a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:20:50.805018 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.804973    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5750adc2-c7a4-466b-a73a-662a71446c4a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5750adc2-c7a4-466b-a73a-662a71446c4a" (UID: "5750adc2-c7a4-466b-a73a-662a71446c4a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:20:50.805081 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.805043    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5750adc2-c7a4-466b-a73a-662a71446c4a-kube-api-access-4rktl" (OuterVolumeSpecName: "kube-api-access-4rktl") pod "5750adc2-c7a4-466b-a73a-662a71446c4a" (UID: "5750adc2-c7a4-466b-a73a-662a71446c4a"). InnerVolumeSpecName "kube-api-access-4rktl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:20:50.904295 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.904222    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5750adc2-c7a4-466b-a73a-662a71446c4a-console-serving-cert\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:20:50.904295 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.904247    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5750adc2-c7a4-466b-a73a-662a71446c4a-console-oauth-config\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:20:50.904295 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.904256    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-trusted-ca-bundle\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:20:50.904295 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.904265    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rktl\" (UniqueName: \"kubernetes.io/projected/5750adc2-c7a4-466b-a73a-662a71446c4a-kube-api-access-4rktl\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:20:50.904295 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.904275    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-oauth-serving-cert\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:20:50.904295 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:50.904284    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5750adc2-c7a4-466b-a73a-662a71446c4a-console-config\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:20:51.164283 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:51.164211    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55ff44d7df-rgr25"
Jul 24 12:20:51.164678 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:51.164207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55ff44d7df-rgr25" event={"ID":"5750adc2-c7a4-466b-a73a-662a71446c4a","Type":"ContainerDied","Data":"2dbc1bf196680adf672039bb0f65228bcfb6221a98e55b3fa592a79680fedd71"}
Jul 24 12:20:51.164678 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:51.164336    2581 scope.go:117] "RemoveContainer" containerID="342fdcebc5ffb1c9a6b224d48895a6128f534208e7c8c9dfcb61974f9aeccd97"
Jul 24 12:20:51.183845 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:51.183824    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55ff44d7df-rgr25"]
Jul 24 12:20:51.187222 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:51.187202    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55ff44d7df-rgr25"]
Jul 24 12:20:52.645747 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:20:52.645713    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5750adc2-c7a4-466b-a73a-662a71446c4a" path="/var/lib/kubelet/pods/5750adc2-c7a4-466b-a73a-662a71446c4a/volumes"
Jul 24 12:21:00.109548 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.109507    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57f58d6854-4j9wh" podUID="b250d142-2d1e-40d5-bf3a-73fa89ae42dc" containerName="console" containerID="cri-o://996067b6e4e7ef8cb70b69ae0cba6f0f38fc14df4addbbb52c868fed89ab9ae0" gracePeriod=15
Jul 24 12:21:00.372853 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.372794    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:21:00.490760 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.490722    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-console-serving-cert\") pod \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") "
Jul 24 12:21:00.490958 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.490793    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-console-oauth-config\") pod \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") "
Jul 24 12:21:00.490958 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.490824    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rh6vt\" (UniqueName: \"kubernetes.io/projected/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-kube-api-access-rh6vt\") pod \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") "
Jul 24 12:21:00.490958 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.490840    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-trusted-ca-bundle\") pod \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") "
Jul 24 12:21:00.490958 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.490869    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-oauth-serving-cert\") pod \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") "
Jul 24 12:21:00.490958 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.490906    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-service-ca\") pod \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") "
Jul 24 12:21:00.491221 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.491049    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-console-config\") pod \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\" (UID: \"b250d142-2d1e-40d5-bf3a-73fa89ae42dc\") "
Jul 24 12:21:00.491380 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.491322    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-service-ca" (OuterVolumeSpecName: "service-ca") pod "b250d142-2d1e-40d5-bf3a-73fa89ae42dc" (UID: "b250d142-2d1e-40d5-bf3a-73fa89ae42dc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:21:00.491380 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.491342    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b250d142-2d1e-40d5-bf3a-73fa89ae42dc" (UID: "b250d142-2d1e-40d5-bf3a-73fa89ae42dc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:21:00.491380 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.491347    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b250d142-2d1e-40d5-bf3a-73fa89ae42dc" (UID: "b250d142-2d1e-40d5-bf3a-73fa89ae42dc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:21:00.491822 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.491545    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-console-config" (OuterVolumeSpecName: "console-config") pod "b250d142-2d1e-40d5-bf3a-73fa89ae42dc" (UID: "b250d142-2d1e-40d5-bf3a-73fa89ae42dc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:21:00.491822 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.491598    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-trusted-ca-bundle\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:21:00.491822 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.491617    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-oauth-serving-cert\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:21:00.491822 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.491631    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-service-ca\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:21:00.493105 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.493075    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b250d142-2d1e-40d5-bf3a-73fa89ae42dc" (UID: "b250d142-2d1e-40d5-bf3a-73fa89ae42dc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:21:00.493227 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.493209    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-kube-api-access-rh6vt" (OuterVolumeSpecName: "kube-api-access-rh6vt") pod "b250d142-2d1e-40d5-bf3a-73fa89ae42dc" (UID: "b250d142-2d1e-40d5-bf3a-73fa89ae42dc"). InnerVolumeSpecName "kube-api-access-rh6vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:21:00.493297 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.493275    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b250d142-2d1e-40d5-bf3a-73fa89ae42dc" (UID: "b250d142-2d1e-40d5-bf3a-73fa89ae42dc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:21:00.592217 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.592185    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-console-serving-cert\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:21:00.592217 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.592210    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-console-oauth-config\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:21:00.592217 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.592221    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh6vt\" (UniqueName: \"kubernetes.io/projected/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-kube-api-access-rh6vt\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:21:00.592487 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:00.592231    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b250d142-2d1e-40d5-bf3a-73fa89ae42dc-console-config\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:21:01.211610 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:01.211566    2581 generic.go:356] "Generic (PLEG): container finished" podID="b250d142-2d1e-40d5-bf3a-73fa89ae42dc" containerID="996067b6e4e7ef8cb70b69ae0cba6f0f38fc14df4addbbb52c868fed89ab9ae0" exitCode=0
Jul 24 12:21:01.212113 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:01.211706    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f58d6854-4j9wh" event={"ID":"b250d142-2d1e-40d5-bf3a-73fa89ae42dc","Type":"ContainerDied","Data":"996067b6e4e7ef8cb70b69ae0cba6f0f38fc14df4addbbb52c868fed89ab9ae0"}
Jul 24 12:21:01.212113 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:01.211782    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f58d6854-4j9wh" event={"ID":"b250d142-2d1e-40d5-bf3a-73fa89ae42dc","Type":"ContainerDied","Data":"ee7c4acda11fe3e8051112ec19bb49238bed5d0a3e81e10a381342503b2d1b52"}
Jul 24 12:21:01.212113 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:01.211817    2581 scope.go:117] "RemoveContainer" containerID="996067b6e4e7ef8cb70b69ae0cba6f0f38fc14df4addbbb52c868fed89ab9ae0"
Jul 24 12:21:01.212113 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:01.212008    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f58d6854-4j9wh"
Jul 24 12:21:01.221019 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:01.221001    2581 scope.go:117] "RemoveContainer" containerID="996067b6e4e7ef8cb70b69ae0cba6f0f38fc14df4addbbb52c868fed89ab9ae0"
Jul 24 12:21:01.221286 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:21:01.221264    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"996067b6e4e7ef8cb70b69ae0cba6f0f38fc14df4addbbb52c868fed89ab9ae0\": container with ID starting with 996067b6e4e7ef8cb70b69ae0cba6f0f38fc14df4addbbb52c868fed89ab9ae0 not found: ID does not exist" containerID="996067b6e4e7ef8cb70b69ae0cba6f0f38fc14df4addbbb52c868fed89ab9ae0"
Jul 24 12:21:01.221331 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:01.221296    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"996067b6e4e7ef8cb70b69ae0cba6f0f38fc14df4addbbb52c868fed89ab9ae0"} err="failed to get container status \"996067b6e4e7ef8cb70b69ae0cba6f0f38fc14df4addbbb52c868fed89ab9ae0\": rpc error: code = NotFound desc = could not find container \"996067b6e4e7ef8cb70b69ae0cba6f0f38fc14df4addbbb52c868fed89ab9ae0\": container with ID starting with 996067b6e4e7ef8cb70b69ae0cba6f0f38fc14df4addbbb52c868fed89ab9ae0 not found: ID does not exist"
Jul 24 12:21:01.238535 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:01.238501    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57f58d6854-4j9wh"]
Jul 24 12:21:01.246008 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:01.245981    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57f58d6854-4j9wh"]
Jul 24 12:21:02.645059 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:02.645027    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b250d142-2d1e-40d5-bf3a-73fa89ae42dc" path="/var/lib/kubelet/pods/b250d142-2d1e-40d5-bf3a-73fa89ae42dc/volumes"
Jul 24 12:21:03.251208 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:03.251178    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:21:03.254915 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:03.254893    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-86866dff5f-8ph8r"
Jul 24 12:21:10.116299 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:10.116267    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8294s"
Jul 24 12:21:21.238596 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:21.238550    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:21:21.258831 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:21.258807    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:21:21.326626 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:21.326600    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 12:21:42.989153 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:21:42.989114    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bf94dd8db-zgnxm"]
Jul 24 12:22:08.008303 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.008261    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6bf94dd8db-zgnxm" podUID="7651e1da-a87d-4427-982d-b588b77dddab" containerName="console" containerID="cri-o://feb5a0ff65b830195e490721872367dd62a00c225d8a4452a31565244470f047" gracePeriod=15
Jul 24 12:22:08.254911 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.254887    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:22:08.314768 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.314740    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2w9r\" (UniqueName: \"kubernetes.io/projected/7651e1da-a87d-4427-982d-b588b77dddab-kube-api-access-b2w9r\") pod \"7651e1da-a87d-4427-982d-b588b77dddab\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") "
Jul 24 12:22:08.314945 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.314796    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-trusted-ca-bundle\") pod \"7651e1da-a87d-4427-982d-b588b77dddab\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") "
Jul 24 12:22:08.314945 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.314825    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7651e1da-a87d-4427-982d-b588b77dddab-console-serving-cert\") pod \"7651e1da-a87d-4427-982d-b588b77dddab\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") "
Jul 24 12:22:08.314945 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.314845    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7651e1da-a87d-4427-982d-b588b77dddab-console-oauth-config\") pod \"7651e1da-a87d-4427-982d-b588b77dddab\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") "
Jul 24 12:22:08.314945 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.314873    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-console-config\") pod \"7651e1da-a87d-4427-982d-b588b77dddab\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") "
Jul 24 12:22:08.315161 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.314989    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-oauth-serving-cert\") pod \"7651e1da-a87d-4427-982d-b588b77dddab\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") "
Jul 24 12:22:08.315161 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.315043    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-service-ca\") pod \"7651e1da-a87d-4427-982d-b588b77dddab\" (UID: \"7651e1da-a87d-4427-982d-b588b77dddab\") "
Jul 24 12:22:08.315311 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.315283    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-console-config" (OuterVolumeSpecName: "console-config") pod "7651e1da-a87d-4427-982d-b588b77dddab" (UID: "7651e1da-a87d-4427-982d-b588b77dddab"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:22:08.315390 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.315307    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7651e1da-a87d-4427-982d-b588b77dddab" (UID: "7651e1da-a87d-4427-982d-b588b77dddab"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:22:08.315390 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.315322    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7651e1da-a87d-4427-982d-b588b77dddab" (UID: "7651e1da-a87d-4427-982d-b588b77dddab"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:22:08.315390 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.315374    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-console-config\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:22:08.315831 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.315802    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-service-ca" (OuterVolumeSpecName: "service-ca") pod "7651e1da-a87d-4427-982d-b588b77dddab" (UID: "7651e1da-a87d-4427-982d-b588b77dddab"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:22:08.317205 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.317175    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7651e1da-a87d-4427-982d-b588b77dddab-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7651e1da-a87d-4427-982d-b588b77dddab" (UID: "7651e1da-a87d-4427-982d-b588b77dddab"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:22:08.317205 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.317193    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7651e1da-a87d-4427-982d-b588b77dddab-kube-api-access-b2w9r" (OuterVolumeSpecName: "kube-api-access-b2w9r") pod "7651e1da-a87d-4427-982d-b588b77dddab" (UID: "7651e1da-a87d-4427-982d-b588b77dddab"). InnerVolumeSpecName "kube-api-access-b2w9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:22:08.317310 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.317239    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7651e1da-a87d-4427-982d-b588b77dddab-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7651e1da-a87d-4427-982d-b588b77dddab" (UID: "7651e1da-a87d-4427-982d-b588b77dddab"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:22:08.416119 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.416080    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-oauth-serving-cert\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:22:08.416119 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.416110    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-service-ca\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:22:08.416119 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.416120    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2w9r\" (UniqueName: \"kubernetes.io/projected/7651e1da-a87d-4427-982d-b588b77dddab-kube-api-access-b2w9r\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:22:08.416349 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.416130    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7651e1da-a87d-4427-982d-b588b77dddab-trusted-ca-bundle\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:22:08.416349 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.416139    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7651e1da-a87d-4427-982d-b588b77dddab-console-serving-cert\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:22:08.416349 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.416148    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7651e1da-a87d-4427-982d-b588b77dddab-console-oauth-config\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:22:08.506020 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.505987    2581 generic.go:356] "Generic (PLEG): container finished" podID="7651e1da-a87d-4427-982d-b588b77dddab" containerID="feb5a0ff65b830195e490721872367dd62a00c225d8a4452a31565244470f047" exitCode=0
Jul 24 12:22:08.506173 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.506035    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bf94dd8db-zgnxm"
Jul 24 12:22:08.506173 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.506072    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bf94dd8db-zgnxm" event={"ID":"7651e1da-a87d-4427-982d-b588b77dddab","Type":"ContainerDied","Data":"feb5a0ff65b830195e490721872367dd62a00c225d8a4452a31565244470f047"}
Jul 24 12:22:08.506173 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.506123    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bf94dd8db-zgnxm" event={"ID":"7651e1da-a87d-4427-982d-b588b77dddab","Type":"ContainerDied","Data":"784591f75ec7ff56aca10f8d78d8651252035501abe20e543edead01fcd46d46"}
Jul 24 12:22:08.506173 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.506143    2581 scope.go:117] "RemoveContainer" containerID="feb5a0ff65b830195e490721872367dd62a00c225d8a4452a31565244470f047"
Jul 24 12:22:08.515311 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.515288    2581 scope.go:117] "RemoveContainer" containerID="feb5a0ff65b830195e490721872367dd62a00c225d8a4452a31565244470f047"
Jul 24 12:22:08.515611 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:22:08.515592    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb5a0ff65b830195e490721872367dd62a00c225d8a4452a31565244470f047\": container with ID starting with feb5a0ff65b830195e490721872367dd62a00c225d8a4452a31565244470f047 not found: ID does not exist" containerID="feb5a0ff65b830195e490721872367dd62a00c225d8a4452a31565244470f047"
Jul 24 12:22:08.515664 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.515620    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb5a0ff65b830195e490721872367dd62a00c225d8a4452a31565244470f047"} err="failed to get container status \"feb5a0ff65b830195e490721872367dd62a00c225d8a4452a31565244470f047\": rpc error: code = NotFound desc = could not find container \"feb5a0ff65b830195e490721872367dd62a00c225d8a4452a31565244470f047\": container with ID starting with feb5a0ff65b830195e490721872367dd62a00c225d8a4452a31565244470f047 not found: ID does not exist"
Jul 24 12:22:08.527165 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.527144    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bf94dd8db-zgnxm"]
Jul 24 12:22:08.531029 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.531010    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bf94dd8db-zgnxm"]
Jul 24 12:22:08.649559 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:22:08.649483    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7651e1da-a87d-4427-982d-b588b77dddab" path="/var/lib/kubelet/pods/7651e1da-a87d-4427-982d-b588b77dddab/volumes"
Jul 24 12:23:05.230185 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.230150    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-r9smg"]
Jul 24 12:23:05.230662 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.230527    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7651e1da-a87d-4427-982d-b588b77dddab" containerName="console"
Jul 24 12:23:05.230662 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.230540    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7651e1da-a87d-4427-982d-b588b77dddab" containerName="console"
Jul 24 12:23:05.230662 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.230555    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b250d142-2d1e-40d5-bf3a-73fa89ae42dc" containerName="console"
Jul 24 12:23:05.230662 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.230560    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b250d142-2d1e-40d5-bf3a-73fa89ae42dc" containerName="console"
Jul 24 12:23:05.230662 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.230573    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5750adc2-c7a4-466b-a73a-662a71446c4a" containerName="console"
Jul 24 12:23:05.230662 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.230579    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5750adc2-c7a4-466b-a73a-662a71446c4a" containerName="console"
Jul 24 12:23:05.230662 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.230634    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b250d142-2d1e-40d5-bf3a-73fa89ae42dc" containerName="console"
Jul 24 12:23:05.230662 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.230642    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5750adc2-c7a4-466b-a73a-662a71446c4a" containerName="console"
Jul 24 12:23:05.230662 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.230649    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7651e1da-a87d-4427-982d-b588b77dddab" containerName="console"
Jul 24 12:23:05.233470 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.233452    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9smg"
Jul 24 12:23:05.236123 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.236100    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-prj5d\""
Jul 24 12:23:05.236595 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.236576    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 24 12:23:05.236652 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.236620    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 24 12:23:05.241958 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.241936    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-r9smg"]
Jul 24 12:23:05.333113 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.333074    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjdd8\" (UniqueName: \"kubernetes.io/projected/e4eb160d-8393-4dc5-a821-15e0e17de325-kube-api-access-cjdd8\") pod \"cert-manager-cainjector-559cdd8c87-r9smg\" (UID: \"e4eb160d-8393-4dc5-a821-15e0e17de325\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9smg"
Jul 24 12:23:05.333280 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.333134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4eb160d-8393-4dc5-a821-15e0e17de325-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-r9smg\" (UID: \"e4eb160d-8393-4dc5-a821-15e0e17de325\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9smg"
Jul 24 12:23:05.434601 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.434489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4eb160d-8393-4dc5-a821-15e0e17de325-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-r9smg\" (UID: \"e4eb160d-8393-4dc5-a821-15e0e17de325\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9smg"
Jul 24 12:23:05.434601 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.434568    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjdd8\" (UniqueName: \"kubernetes.io/projected/e4eb160d-8393-4dc5-a821-15e0e17de325-kube-api-access-cjdd8\") pod \"cert-manager-cainjector-559cdd8c87-r9smg\" (UID: \"e4eb160d-8393-4dc5-a821-15e0e17de325\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9smg"
Jul 24 12:23:05.442721 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.442689    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4eb160d-8393-4dc5-a821-15e0e17de325-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-r9smg\" (UID: \"e4eb160d-8393-4dc5-a821-15e0e17de325\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9smg"
Jul 24 12:23:05.442860 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.442837    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjdd8\" (UniqueName: \"kubernetes.io/projected/e4eb160d-8393-4dc5-a821-15e0e17de325-kube-api-access-cjdd8\") pod \"cert-manager-cainjector-559cdd8c87-r9smg\" (UID: \"e4eb160d-8393-4dc5-a821-15e0e17de325\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9smg"
Jul 24 12:23:05.552097 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.552056    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9smg"
Jul 24 12:23:05.686473 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:23:05.686435    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4eb160d_8393_4dc5_a821_15e0e17de325.slice/crio-0f020b12cc7239f64940c2117773d6d82d59674258f68f82e88325141c1f569f WatchSource:0}: Error finding container 0f020b12cc7239f64940c2117773d6d82d59674258f68f82e88325141c1f569f: Status 404 returned error can't find the container with id 0f020b12cc7239f64940c2117773d6d82d59674258f68f82e88325141c1f569f
Jul 24 12:23:05.687582 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.687562    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-r9smg"]
Jul 24 12:23:05.735542 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:05.735505    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9smg" event={"ID":"e4eb160d-8393-4dc5-a821-15e0e17de325","Type":"ContainerStarted","Data":"0f020b12cc7239f64940c2117773d6d82d59674258f68f82e88325141c1f569f"}
Jul 24 12:23:09.755481 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:09.755445    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9smg" event={"ID":"e4eb160d-8393-4dc5-a821-15e0e17de325","Type":"ContainerStarted","Data":"0aa731653bc6603272cb573d150cf41478087e3fcb9034a80d8ef54c8cae55bc"}
Jul 24 12:23:09.771022 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:09.770970    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r9smg" podStartSLOduration=1.542718386 podStartE2EDuration="4.770951192s" totalImagesPullingTime="3.228232806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:23:09.770694004 +0000 UTC m=+221.725698767" watchObservedRunningTime="2026-07-24 12:23:09.770951192 +0000 UTC m=+221.725955956"
Jul 24 12:23:15.447384 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:15.447350    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck7q4"]
Jul 24 12:23:15.450942 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:15.450924    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck7q4"
Jul 24 12:23:15.453096 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:15.453070    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 24 12:23:15.453200 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:15.453133    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-zvmcn\""
Jul 24 12:23:15.453614 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:15.453597    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 24 12:23:15.463495 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:15.463474    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck7q4"]
Jul 24 12:23:15.629788 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:15.629738    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a282e7d-5bab-4d0d-b155-80e2d435804d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ck7q4\" (UID: \"3a282e7d-5bab-4d0d-b155-80e2d435804d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck7q4"
Jul 24 12:23:15.629788 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:15.629786    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft2h6\" (UniqueName: \"kubernetes.io/projected/3a282e7d-5bab-4d0d-b155-80e2d435804d-kube-api-access-ft2h6\") pod \"openshift-lws-operator-fd8ccff4c-ck7q4\" (UID: \"3a282e7d-5bab-4d0d-b155-80e2d435804d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck7q4"
Jul 24 12:23:15.731043 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:15.730960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a282e7d-5bab-4d0d-b155-80e2d435804d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ck7q4\" (UID: \"3a282e7d-5bab-4d0d-b155-80e2d435804d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck7q4"
Jul 24 12:23:15.731043 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:15.730998    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft2h6\" (UniqueName: \"kubernetes.io/projected/3a282e7d-5bab-4d0d-b155-80e2d435804d-kube-api-access-ft2h6\") pod \"openshift-lws-operator-fd8ccff4c-ck7q4\" (UID: \"3a282e7d-5bab-4d0d-b155-80e2d435804d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck7q4"
Jul 24 12:23:15.731445 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:15.731398    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a282e7d-5bab-4d0d-b155-80e2d435804d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ck7q4\" (UID: \"3a282e7d-5bab-4d0d-b155-80e2d435804d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck7q4"
Jul 24 12:23:15.739350 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:15.739329    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft2h6\" (UniqueName: \"kubernetes.io/projected/3a282e7d-5bab-4d0d-b155-80e2d435804d-kube-api-access-ft2h6\") pod \"openshift-lws-operator-fd8ccff4c-ck7q4\" (UID: \"3a282e7d-5bab-4d0d-b155-80e2d435804d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck7q4"
Jul 24 12:23:15.760311 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:15.760274    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck7q4"
Jul 24 12:23:15.886933 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:15.886868    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck7q4"]
Jul 24 12:23:15.889725 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:23:15.889698    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a282e7d_5bab_4d0d_b155_80e2d435804d.slice/crio-2755af146dd2c644576b54470be77f801ae35c8e96a46eb20c5bb3dce1be5ecd WatchSource:0}: Error finding container 2755af146dd2c644576b54470be77f801ae35c8e96a46eb20c5bb3dce1be5ecd: Status 404 returned error can't find the container with id 2755af146dd2c644576b54470be77f801ae35c8e96a46eb20c5bb3dce1be5ecd
Jul 24 12:23:16.796926 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:16.796889    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck7q4" event={"ID":"3a282e7d-5bab-4d0d-b155-80e2d435804d","Type":"ContainerStarted","Data":"2755af146dd2c644576b54470be77f801ae35c8e96a46eb20c5bb3dce1be5ecd"}
Jul 24 12:23:25.836560 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:25.836519    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck7q4" event={"ID":"3a282e7d-5bab-4d0d-b155-80e2d435804d","Type":"ContainerStarted","Data":"8eea4191e43c43c45b80c03bb9e20bab6da17e09f0b80aa326c8d0a9b5286b96"}
Jul 24 12:23:25.855620 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:25.855391    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck7q4" podStartSLOduration=1.5503108349999999 podStartE2EDuration="10.855370477s" totalImagesPullingTime="9.305059642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:23:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:23:25.854755912 +0000 UTC m=+237.809760677" watchObservedRunningTime="2026-07-24 12:23:25.855370477 +0000 UTC m=+237.810375243"
Jul 24 12:23:37.030045 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.030010    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6"]
Jul 24 12:23:37.033493 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.033475    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6"
Jul 24 12:23:37.035780 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.035749    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ggthh\""
Jul 24 12:23:37.036379 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.036360    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 24 12:23:37.036682 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.036668    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 24 12:23:37.048490 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.048468    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 24 12:23:37.057967 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.057945    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6"]
Jul 24 12:23:37.122650 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.122611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c46d08c-90dd-4a01-8a5f-fd8985444c68-cert\") pod \"lws-controller-manager-d4b69f9c5-9h6f6\" (UID: \"2c46d08c-90dd-4a01-8a5f-fd8985444c68\") " pod="openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6"
Jul 24 12:23:37.122650 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.122654    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2c46d08c-90dd-4a01-8a5f-fd8985444c68-manager-config\") pod \"lws-controller-manager-d4b69f9c5-9h6f6\" (UID: \"2c46d08c-90dd-4a01-8a5f-fd8985444c68\") " pod="openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6"
Jul 24 12:23:37.122863 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.122678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c46d08c-90dd-4a01-8a5f-fd8985444c68-metrics-cert\") pod \"lws-controller-manager-d4b69f9c5-9h6f6\" (UID: \"2c46d08c-90dd-4a01-8a5f-fd8985444c68\") " pod="openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6"
Jul 24 12:23:37.122863 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.122766    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mg2p\" (UniqueName: \"kubernetes.io/projected/2c46d08c-90dd-4a01-8a5f-fd8985444c68-kube-api-access-4mg2p\") pod \"lws-controller-manager-d4b69f9c5-9h6f6\" (UID: \"2c46d08c-90dd-4a01-8a5f-fd8985444c68\") " pod="openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6"
Jul 24 12:23:37.223551 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.223507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c46d08c-90dd-4a01-8a5f-fd8985444c68-cert\") pod \"lws-controller-manager-d4b69f9c5-9h6f6\" (UID: \"2c46d08c-90dd-4a01-8a5f-fd8985444c68\") " pod="openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6"
Jul 24 12:23:37.223551 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.223551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2c46d08c-90dd-4a01-8a5f-fd8985444c68-manager-config\") pod \"lws-controller-manager-d4b69f9c5-9h6f6\" (UID: \"2c46d08c-90dd-4a01-8a5f-fd8985444c68\") " pod="openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6"
Jul 24 12:23:37.223810 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.223576    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c46d08c-90dd-4a01-8a5f-fd8985444c68-metrics-cert\") pod \"lws-controller-manager-d4b69f9c5-9h6f6\" (UID: \"2c46d08c-90dd-4a01-8a5f-fd8985444c68\") " pod="openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6"
Jul 24 12:23:37.223810 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.223628    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mg2p\" (UniqueName: \"kubernetes.io/projected/2c46d08c-90dd-4a01-8a5f-fd8985444c68-kube-api-access-4mg2p\") pod \"lws-controller-manager-d4b69f9c5-9h6f6\" (UID: \"2c46d08c-90dd-4a01-8a5f-fd8985444c68\") " pod="openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6"
Jul 24 12:23:37.224153 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.224129    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2c46d08c-90dd-4a01-8a5f-fd8985444c68-manager-config\") pod \"lws-controller-manager-d4b69f9c5-9h6f6\" (UID: \"2c46d08c-90dd-4a01-8a5f-fd8985444c68\") " pod="openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6"
Jul 24 12:23:37.226166 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.226143    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c46d08c-90dd-4a01-8a5f-fd8985444c68-cert\") pod \"lws-controller-manager-d4b69f9c5-9h6f6\" (UID: \"2c46d08c-90dd-4a01-8a5f-fd8985444c68\") " pod="openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6"
Jul 24 12:23:37.226249 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.226219    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c46d08c-90dd-4a01-8a5f-fd8985444c68-metrics-cert\") pod \"lws-controller-manager-d4b69f9c5-9h6f6\" (UID: \"2c46d08c-90dd-4a01-8a5f-fd8985444c68\") " pod="openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6"
Jul 24 12:23:37.235368 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.235337    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mg2p\" (UniqueName: \"kubernetes.io/projected/2c46d08c-90dd-4a01-8a5f-fd8985444c68-kube-api-access-4mg2p\") pod \"lws-controller-manager-d4b69f9c5-9h6f6\" (UID: \"2c46d08c-90dd-4a01-8a5f-fd8985444c68\") " pod="openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6"
Jul 24 12:23:37.342492 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.342458    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6"
Jul 24 12:23:37.502385 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.502352    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6"]
Jul 24 12:23:37.505835 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:23:37.505801    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c46d08c_90dd_4a01_8a5f_fd8985444c68.slice/crio-31290c8d3d8781c009b6d863f95f96669d12653f5751e8e0513c081e41e66a8f WatchSource:0}: Error finding container 31290c8d3d8781c009b6d863f95f96669d12653f5751e8e0513c081e41e66a8f: Status 404 returned error can't find the container with id 31290c8d3d8781c009b6d863f95f96669d12653f5751e8e0513c081e41e66a8f
Jul 24 12:23:37.884350 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:37.884319    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6" event={"ID":"2c46d08c-90dd-4a01-8a5f-fd8985444c68","Type":"ContainerStarted","Data":"31290c8d3d8781c009b6d863f95f96669d12653f5751e8e0513c081e41e66a8f"}
Jul 24 12:23:39.895369 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:39.895326    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6" event={"ID":"2c46d08c-90dd-4a01-8a5f-fd8985444c68","Type":"ContainerStarted","Data":"a98f63eed5ffb6be7f144af3bdc99d2e8e318be214b777869f998a5ec36d28ef"}
Jul 24 12:23:39.895753 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:39.895473    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6"
Jul 24 12:23:46.363907 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.363849    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6" podStartSLOduration=8.717616804 podStartE2EDuration="10.363832935s" totalImagesPullingTime="1.646216131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:23:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:23:39.921143822 +0000 UTC m=+251.876148596" watchObservedRunningTime="2026-07-24 12:23:46.363832935 +0000 UTC m=+258.318837698"
Jul 24 12:23:46.365553 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.365529    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"]
Jul 24 12:23:46.369462 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.369443    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"
Jul 24 12:23:46.372899 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.372878    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 24 12:23:46.373150 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.373134    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-w2xqh\""
Jul 24 12:23:46.374222 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.374203    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 24 12:23:46.376954 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.376938    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 24 12:23:46.377312 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.377296    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 24 12:23:46.403079 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.403046    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"]
Jul 24 12:23:46.504865 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.504834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eff6885f-c754-4a73-a93f-0b9ccd7de582-manifests\") pod \"opendatahub-operator-controller-manager-565c4d787c-v5dfd\" (UID: \"eff6885f-c754-4a73-a93f-0b9ccd7de582\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"
Jul 24 12:23:46.504865 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.504867    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eff6885f-c754-4a73-a93f-0b9ccd7de582-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565c4d787c-v5dfd\" (UID: \"eff6885f-c754-4a73-a93f-0b9ccd7de582\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"
Jul 24 12:23:46.505118 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.504894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eff6885f-c754-4a73-a93f-0b9ccd7de582-webhook-cert\") pod \"opendatahub-operator-controller-manager-565c4d787c-v5dfd\" (UID: \"eff6885f-c754-4a73-a93f-0b9ccd7de582\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"
Jul 24 12:23:46.505118 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.504986    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eff6885f-c754-4a73-a93f-0b9ccd7de582-tmp\") pod \"opendatahub-operator-controller-manager-565c4d787c-v5dfd\" (UID: \"eff6885f-c754-4a73-a93f-0b9ccd7de582\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"
Jul 24 12:23:46.505118 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.505027    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfg55\" (UniqueName: \"kubernetes.io/projected/eff6885f-c754-4a73-a93f-0b9ccd7de582-kube-api-access-vfg55\") pod \"opendatahub-operator-controller-manager-565c4d787c-v5dfd\" (UID: \"eff6885f-c754-4a73-a93f-0b9ccd7de582\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"
Jul 24 12:23:46.605773 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.605735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eff6885f-c754-4a73-a93f-0b9ccd7de582-webhook-cert\") pod \"opendatahub-operator-controller-manager-565c4d787c-v5dfd\" (UID: \"eff6885f-c754-4a73-a93f-0b9ccd7de582\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"
Jul 24 12:23:46.605973 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.605794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eff6885f-c754-4a73-a93f-0b9ccd7de582-tmp\") pod \"opendatahub-operator-controller-manager-565c4d787c-v5dfd\" (UID: \"eff6885f-c754-4a73-a93f-0b9ccd7de582\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"
Jul 24 12:23:46.605973 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.605831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfg55\" (UniqueName: \"kubernetes.io/projected/eff6885f-c754-4a73-a93f-0b9ccd7de582-kube-api-access-vfg55\") pod \"opendatahub-operator-controller-manager-565c4d787c-v5dfd\" (UID: \"eff6885f-c754-4a73-a93f-0b9ccd7de582\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"
Jul 24 12:23:46.605973 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.605921    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eff6885f-c754-4a73-a93f-0b9ccd7de582-manifests\") pod \"opendatahub-operator-controller-manager-565c4d787c-v5dfd\" (UID: \"eff6885f-c754-4a73-a93f-0b9ccd7de582\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"
Jul 24 12:23:46.605973 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.605946    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eff6885f-c754-4a73-a93f-0b9ccd7de582-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565c4d787c-v5dfd\" (UID: \"eff6885f-c754-4a73-a93f-0b9ccd7de582\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"
Jul 24 12:23:46.606309 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.606285    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eff6885f-c754-4a73-a93f-0b9ccd7de582-manifests\") pod \"opendatahub-operator-controller-manager-565c4d787c-v5dfd\" (UID: \"eff6885f-c754-4a73-a93f-0b9ccd7de582\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"
Jul 24 12:23:46.608180 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.608150    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eff6885f-c754-4a73-a93f-0b9ccd7de582-tmp\") pod \"opendatahub-operator-controller-manager-565c4d787c-v5dfd\" (UID: \"eff6885f-c754-4a73-a93f-0b9ccd7de582\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"
Jul 24 12:23:46.608301 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.608257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eff6885f-c754-4a73-a93f-0b9ccd7de582-webhook-cert\") pod \"opendatahub-operator-controller-manager-565c4d787c-v5dfd\" (UID: \"eff6885f-c754-4a73-a93f-0b9ccd7de582\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"
Jul 24 12:23:46.608360 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.608310    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eff6885f-c754-4a73-a93f-0b9ccd7de582-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565c4d787c-v5dfd\" (UID: \"eff6885f-c754-4a73-a93f-0b9ccd7de582\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"
Jul 24 12:23:46.624817 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.624761    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfg55\" (UniqueName: \"kubernetes.io/projected/eff6885f-c754-4a73-a93f-0b9ccd7de582-kube-api-access-vfg55\") pod \"opendatahub-operator-controller-manager-565c4d787c-v5dfd\" (UID: \"eff6885f-c754-4a73-a93f-0b9ccd7de582\") " pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"
Jul 24 12:23:46.678875 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.678840    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"
Jul 24 12:23:46.834252 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.834224    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"]
Jul 24 12:23:46.835546 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:23:46.835523    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff6885f_c754_4a73_a93f_0b9ccd7de582.slice/crio-8282cc29a3e6b0964420dcf9fa6c3fb1c1a63d9a7714f9f76827b956de4b21fa WatchSource:0}: Error finding container 8282cc29a3e6b0964420dcf9fa6c3fb1c1a63d9a7714f9f76827b956de4b21fa: Status 404 returned error can't find the container with id 8282cc29a3e6b0964420dcf9fa6c3fb1c1a63d9a7714f9f76827b956de4b21fa
Jul 24 12:23:46.925923 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:46.925838    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd" event={"ID":"eff6885f-c754-4a73-a93f-0b9ccd7de582","Type":"ContainerStarted","Data":"8282cc29a3e6b0964420dcf9fa6c3fb1c1a63d9a7714f9f76827b956de4b21fa"}
Jul 24 12:23:50.902169 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:50.902138    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-d4b69f9c5-9h6f6"
Jul 24 12:23:50.952478 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:50.952400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd" event={"ID":"eff6885f-c754-4a73-a93f-0b9ccd7de582","Type":"ContainerStarted","Data":"3505d2a85b9afccbf195276cecafa86d06d0bae3c99c264c75f48670498fedcb"}
Jul 24 12:23:51.958996 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:51.958909    2581 generic.go:356] "Generic (PLEG): container finished" podID="eff6885f-c754-4a73-a93f-0b9ccd7de582" containerID="3505d2a85b9afccbf195276cecafa86d06d0bae3c99c264c75f48670498fedcb" exitCode=0
Jul 24 12:23:51.959346 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:51.958997    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd" event={"ID":"eff6885f-c754-4a73-a93f-0b9ccd7de582","Type":"ContainerDied","Data":"3505d2a85b9afccbf195276cecafa86d06d0bae3c99c264c75f48670498fedcb"}
Jul 24 12:23:52.967277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:52.967241    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd" event={"ID":"eff6885f-c754-4a73-a93f-0b9ccd7de582","Type":"ContainerStarted","Data":"a71be2d3a1b0e99695c972cf7c0eabf267c83f5e2a213e254add9ef8532374be"}
Jul 24 12:23:52.967781 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:52.967328    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"
Jul 24 12:23:52.992093 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:23:52.992026    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd" podStartSLOduration=1.97045606 podStartE2EDuration="6.992005145s" totalImagesPullingTime="3.897530305s" totalInitContainerRuntime="1.12401878s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:23:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:23:52.990671906 +0000 UTC m=+264.945676668" watchObservedRunningTime="2026-07-24 12:23:52.992005145 +0000 UTC m=+264.947009902"
Jul 24 12:24:03.974738 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:03.974709    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-565c4d787c-v5dfd"
Jul 24 12:24:18.093575 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:18.093543    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5bfc95c995-v4dz5"]
Jul 24 12:24:18.096772 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:18.096746    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bfc95c995-v4dz5"
Jul 24 12:24:18.103084 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:24:18.103040    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kube-auth-proxy-5bfc95c995-v4dz5\" is forbidden: User \"system:node:ip-10-0-128-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-194.ec2.internal' and this object" podUID="477446d3-4478-4510-a2e7-59d1d88bd115" pod="openshift-ingress/kube-auth-proxy-5bfc95c995-v4dz5"
Jul 24 12:24:18.103212 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:24:18.103146    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-auth-proxy-tls\" is forbidden: User \"system:node:ip-10-0-128-194.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-194.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret"
Jul 24 12:24:18.103212 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:24:18.103146    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-auth-proxy-dockercfg-rtssb\" is forbidden: User \"system:node:ip-10-0-128-194.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-194.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rtssb\"" type="*v1.Secret"
Jul 24 12:24:18.104052 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:24:18.104034    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-194.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-194.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 24 12:24:18.109386 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:18.109371    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 24 12:24:18.151581 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:18.151552    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 24 12:24:18.167747 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:18.167710    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bfc95c995-v4dz5"]
Jul 24 12:24:18.192383 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:18.192347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzjn4\" (UniqueName: \"kubernetes.io/projected/477446d3-4478-4510-a2e7-59d1d88bd115-kube-api-access-jzjn4\") pod \"kube-auth-proxy-5bfc95c995-v4dz5\" (UID: \"477446d3-4478-4510-a2e7-59d1d88bd115\") " pod="openshift-ingress/kube-auth-proxy-5bfc95c995-v4dz5"
Jul 24 12:24:18.192577 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:18.192411    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/477446d3-4478-4510-a2e7-59d1d88bd115-tls-certs\") pod \"kube-auth-proxy-5bfc95c995-v4dz5\" (UID: \"477446d3-4478-4510-a2e7-59d1d88bd115\") " pod="openshift-ingress/kube-auth-proxy-5bfc95c995-v4dz5"
Jul 24 12:24:18.192577 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:18.192456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/477446d3-4478-4510-a2e7-59d1d88bd115-tmp\") pod \"kube-auth-proxy-5bfc95c995-v4dz5\" (UID: \"477446d3-4478-4510-a2e7-59d1d88bd115\") " pod="openshift-ingress/kube-auth-proxy-5bfc95c995-v4dz5"
Jul 24 12:24:18.293600 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:18.293553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzjn4\" (UniqueName: \"kubernetes.io/projected/477446d3-4478-4510-a2e7-59d1d88bd115-kube-api-access-jzjn4\") pod \"kube-auth-proxy-5bfc95c995-v4dz5\" (UID: \"477446d3-4478-4510-a2e7-59d1d88bd115\") " pod="openshift-ingress/kube-auth-proxy-5bfc95c995-v4dz5"
Jul 24 12:24:18.293790 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:18.293640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/477446d3-4478-4510-a2e7-59d1d88bd115-tls-certs\") pod \"kube-auth-proxy-5bfc95c995-v4dz5\" (UID: \"477446d3-4478-4510-a2e7-59d1d88bd115\") " pod="openshift-ingress/kube-auth-proxy-5bfc95c995-v4dz5"
Jul 24 12:24:18.293790 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:18.293662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/477446d3-4478-4510-a2e7-59d1d88bd115-tmp\") pod \"kube-auth-proxy-5bfc95c995-v4dz5\" (UID: \"477446d3-4478-4510-a2e7-59d1d88bd115\") " pod="openshift-ingress/kube-auth-proxy-5bfc95c995-v4dz5"
Jul 24 12:24:18.295993 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:18.295969    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/477446d3-4478-4510-a2e7-59d1d88bd115-tmp\") pod \"kube-auth-proxy-5bfc95c995-v4dz5\" (UID: \"477446d3-4478-4510-a2e7-59d1d88bd115\") " pod="openshift-ingress/kube-auth-proxy-5bfc95c995-v4dz5"
Jul 24 12:24:19.012948 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:19.012915    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 24 12:24:19.016322 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:19.016290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/477446d3-4478-4510-a2e7-59d1d88bd115-tls-certs\") pod \"kube-auth-proxy-5bfc95c995-v4dz5\" (UID: \"477446d3-4478-4510-a2e7-59d1d88bd115\") " pod="openshift-ingress/kube-auth-proxy-5bfc95c995-v4dz5"
Jul 24 12:24:19.328449 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:24:19.328383    2581 projected.go:291] Couldn't get configMap openshift-ingress/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Jul 24 12:24:19.328939 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:24:19.328463    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jzjn4 for pod openshift-ingress/kube-auth-proxy-5bfc95c995-v4dz5: failed to sync configmap cache: timed out waiting for the condition
Jul 24 12:24:19.328939 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:24:19.328561    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/477446d3-4478-4510-a2e7-59d1d88bd115-kube-api-access-jzjn4 podName:477446d3-4478-4510-a2e7-59d1d88bd115 nodeName:}" failed. No retries permitted until 2026-07-24 12:24:19.828535978 +0000 UTC m=+291.783540733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jzjn4" (UniqueName: "kubernetes.io/projected/477446d3-4478-4510-a2e7-59d1d88bd115-kube-api-access-jzjn4") pod "kube-auth-proxy-5bfc95c995-v4dz5" (UID: "477446d3-4478-4510-a2e7-59d1d88bd115") : failed to sync configmap cache: timed out waiting for the condition
Jul 24 12:24:19.449179 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:19.449146    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rtssb\""
Jul 24 12:24:19.608331 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:19.608245    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 24 12:24:19.908631 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:19.908539    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzjn4\" (UniqueName: \"kubernetes.io/projected/477446d3-4478-4510-a2e7-59d1d88bd115-kube-api-access-jzjn4\") pod \"kube-auth-proxy-5bfc95c995-v4dz5\" (UID: \"477446d3-4478-4510-a2e7-59d1d88bd115\") " pod="openshift-ingress/kube-auth-proxy-5bfc95c995-v4dz5"
Jul 24 12:24:19.911349 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:19.911310    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzjn4\" (UniqueName: \"kubernetes.io/projected/477446d3-4478-4510-a2e7-59d1d88bd115-kube-api-access-jzjn4\") pod \"kube-auth-proxy-5bfc95c995-v4dz5\" (UID: \"477446d3-4478-4510-a2e7-59d1d88bd115\") " pod="openshift-ingress/kube-auth-proxy-5bfc95c995-v4dz5"
Jul 24 12:24:20.206285 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:20.206211    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bfc95c995-v4dz5"
Jul 24 12:24:20.365595 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:20.365564    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bfc95c995-v4dz5"]
Jul 24 12:24:20.368849 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:24:20.368809    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477446d3_4478_4510_a2e7_59d1d88bd115.slice/crio-f47a8351151bde96a37a20842955e974570062ed7ca403b6a6fb5fa1116e4c02 WatchSource:0}: Error finding container f47a8351151bde96a37a20842955e974570062ed7ca403b6a6fb5fa1116e4c02: Status 404 returned error can't find the container with id f47a8351151bde96a37a20842955e974570062ed7ca403b6a6fb5fa1116e4c02
Jul 24 12:24:21.098480 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:21.098445    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bfc95c995-v4dz5" event={"ID":"477446d3-4478-4510-a2e7-59d1d88bd115","Type":"ContainerStarted","Data":"f47a8351151bde96a37a20842955e974570062ed7ca403b6a6fb5fa1116e4c02"}
Jul 24 12:24:23.111230 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:23.111140    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bfc95c995-v4dz5" event={"ID":"477446d3-4478-4510-a2e7-59d1d88bd115","Type":"ContainerStarted","Data":"e0aff068789af523f01fb1537012f5bd38c9dd6050252252919e63304436c058"}
Jul 24 12:24:23.140867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:23.140816    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5bfc95c995-v4dz5" podStartSLOduration=2.664695948 podStartE2EDuration="5.140797878s" totalImagesPullingTime="2.47610193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:24:23.140667748 +0000 UTC m=+295.095672511" watchObservedRunningTime="2026-07-24 12:24:23.140797878 +0000 UTC m=+295.095802641"
Jul 24 12:24:28.475916 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:24:28.475890    2581 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 12:25:47.282837 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:25:47.282801    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj"]
Jul 24 12:25:47.286272 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:25:47.286254    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj"
Jul 24 12:25:47.292272 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:25:47.292243    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 12:25:47.295567 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:25:47.292695    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vgv4q\""
Jul 24 12:25:47.295567 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:25:47.292742    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 12:25:47.401746 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:25:47.401713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d0427dc-6080-4116-b38b-90130f37c514-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kwfwj\" (UID: \"0d0427dc-6080-4116-b38b-90130f37c514\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj"
Jul 24 12:25:47.401746 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:25:47.401751    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxh4g\" (UniqueName: \"kubernetes.io/projected/0d0427dc-6080-4116-b38b-90130f37c514-kube-api-access-dxh4g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kwfwj\" (UID: \"0d0427dc-6080-4116-b38b-90130f37c514\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj"
Jul 24 12:25:47.412743 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:25:47.412714    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj"]
Jul 24 12:25:47.502857 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:25:47.502820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d0427dc-6080-4116-b38b-90130f37c514-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kwfwj\" (UID: \"0d0427dc-6080-4116-b38b-90130f37c514\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj"
Jul 24 12:25:47.502857 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:25:47.502858    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxh4g\" (UniqueName: \"kubernetes.io/projected/0d0427dc-6080-4116-b38b-90130f37c514-kube-api-access-dxh4g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kwfwj\" (UID: \"0d0427dc-6080-4116-b38b-90130f37c514\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj"
Jul 24 12:25:47.503195 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:25:47.503178    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d0427dc-6080-4116-b38b-90130f37c514-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kwfwj\" (UID: \"0d0427dc-6080-4116-b38b-90130f37c514\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj"
Jul 24 12:25:47.514568 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:25:47.514537    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxh4g\" (UniqueName: \"kubernetes.io/projected/0d0427dc-6080-4116-b38b-90130f37c514-kube-api-access-dxh4g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kwfwj\" (UID: \"0d0427dc-6080-4116-b38b-90130f37c514\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj"
Jul 24 12:25:47.602256 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:25:47.602165    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj"
Jul 24 12:25:47.756948 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:25:47.756920    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj"]
Jul 24 12:25:47.762347 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:25:47.762317    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0427dc_6080_4116_b38b_90130f37c514.slice/crio-9f655b79846ceb20e310b30834a799e4f2038fffb9dc21e450e7e4700cde2486 WatchSource:0}: Error finding container 9f655b79846ceb20e310b30834a799e4f2038fffb9dc21e450e7e4700cde2486: Status 404 returned error can't find the container with id 9f655b79846ceb20e310b30834a799e4f2038fffb9dc21e450e7e4700cde2486
Jul 24 12:25:47.765250 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:25:47.765232    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 12:25:48.499023 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:25:48.498991    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj" event={"ID":"0d0427dc-6080-4116-b38b-90130f37c514","Type":"ContainerStarted","Data":"9f655b79846ceb20e310b30834a799e4f2038fffb9dc21e450e7e4700cde2486"}
Jul 24 12:25:52.521380 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:25:52.521345    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj" event={"ID":"0d0427dc-6080-4116-b38b-90130f37c514","Type":"ContainerStarted","Data":"79eee77756465bab7edfebe237e895d3a350a40afbb2ef71a2782035aed37ead"}
Jul 24 12:25:52.521809 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:25:52.521476    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj"
Jul 24 12:25:52.550725 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:25:52.550654    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj" podStartSLOduration=0.947738199 podStartE2EDuration="5.550434644s" totalImagesPullingTime="4.602696445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:25:52.542160301 +0000 UTC m=+384.497165064" watchObservedRunningTime="2026-07-24 12:25:52.550434644 +0000 UTC m=+384.505439406"
Jul 24 12:26:03.528400 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:03.528366    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj"
Jul 24 12:26:05.437633 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.437586    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj"]
Jul 24 12:26:05.438020 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.437827    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj" podUID="0d0427dc-6080-4116-b38b-90130f37c514" containerName="manager" containerID="cri-o://79eee77756465bab7edfebe237e895d3a350a40afbb2ef71a2782035aed37ead" gracePeriod=2
Jul 24 12:26:05.445370 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.445339    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj"]
Jul 24 12:26:05.473693 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.473669    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx"]
Jul 24 12:26:05.474030 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.474018    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d0427dc-6080-4116-b38b-90130f37c514" containerName="manager"
Jul 24 12:26:05.474088 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.474032    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d0427dc-6080-4116-b38b-90130f37c514" containerName="manager"
Jul 24 12:26:05.474124 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.474092    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d0427dc-6080-4116-b38b-90130f37c514" containerName="manager"
Jul 24 12:26:05.482368 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.482332    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx"
Jul 24 12:26:05.484545 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:26:05.484512    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kwfwj\" is forbidden: User \"system:node:ip-10-0-128-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-194.ec2.internal' and this object" podUID="0d0427dc-6080-4116-b38b-90130f37c514" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj"
Jul 24 12:26:05.501069 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.501043    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx"]
Jul 24 12:26:05.573641 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.573612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qqk5\" (UniqueName: \"kubernetes.io/projected/e1bfdfe2-e95e-41a5-845b-99b7bc047ce7-kube-api-access-7qqk5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bc9sx\" (UID: \"e1bfdfe2-e95e-41a5-845b-99b7bc047ce7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx"
Jul 24 12:26:05.573760 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.573679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1bfdfe2-e95e-41a5-845b-99b7bc047ce7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bc9sx\" (UID: \"e1bfdfe2-e95e-41a5-845b-99b7bc047ce7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx"
Jul 24 12:26:05.580905 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.580877    2581 generic.go:356] "Generic (PLEG): container finished" podID="0d0427dc-6080-4116-b38b-90130f37c514" containerID="79eee77756465bab7edfebe237e895d3a350a40afbb2ef71a2782035aed37ead" exitCode=0
Jul 24 12:26:05.674994 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.674967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qqk5\" (UniqueName: \"kubernetes.io/projected/e1bfdfe2-e95e-41a5-845b-99b7bc047ce7-kube-api-access-7qqk5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bc9sx\" (UID: \"e1bfdfe2-e95e-41a5-845b-99b7bc047ce7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx"
Jul 24 12:26:05.675104 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.675026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1bfdfe2-e95e-41a5-845b-99b7bc047ce7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bc9sx\" (UID: \"e1bfdfe2-e95e-41a5-845b-99b7bc047ce7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx"
Jul 24 12:26:05.675354 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.675338    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1bfdfe2-e95e-41a5-845b-99b7bc047ce7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bc9sx\" (UID: \"e1bfdfe2-e95e-41a5-845b-99b7bc047ce7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx"
Jul 24 12:26:05.681309 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.681291    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj"
Jul 24 12:26:05.683239 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:26:05.683218    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kwfwj\" is forbidden: User \"system:node:ip-10-0-128-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-194.ec2.internal' and this object" podUID="0d0427dc-6080-4116-b38b-90130f37c514" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj"
Jul 24 12:26:05.692571 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.692528    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qqk5\" (UniqueName: \"kubernetes.io/projected/e1bfdfe2-e95e-41a5-845b-99b7bc047ce7-kube-api-access-7qqk5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bc9sx\" (UID: \"e1bfdfe2-e95e-41a5-845b-99b7bc047ce7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx"
Jul 24 12:26:05.775642 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.775609    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxh4g\" (UniqueName: \"kubernetes.io/projected/0d0427dc-6080-4116-b38b-90130f37c514-kube-api-access-dxh4g\") pod \"0d0427dc-6080-4116-b38b-90130f37c514\" (UID: \"0d0427dc-6080-4116-b38b-90130f37c514\") "
Jul 24 12:26:05.775805 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.775700    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d0427dc-6080-4116-b38b-90130f37c514-extensions-socket-volume\") pod \"0d0427dc-6080-4116-b38b-90130f37c514\" (UID: \"0d0427dc-6080-4116-b38b-90130f37c514\") "
Jul 24 12:26:05.776157 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.776131    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0427dc-6080-4116-b38b-90130f37c514-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0d0427dc-6080-4116-b38b-90130f37c514" (UID: "0d0427dc-6080-4116-b38b-90130f37c514"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:26:05.777886 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.777863    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0427dc-6080-4116-b38b-90130f37c514-kube-api-access-dxh4g" (OuterVolumeSpecName: "kube-api-access-dxh4g") pod "0d0427dc-6080-4116-b38b-90130f37c514" (UID: "0d0427dc-6080-4116-b38b-90130f37c514"). InnerVolumeSpecName "kube-api-access-dxh4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:26:05.839739 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.839696    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx"
Jul 24 12:26:05.877435 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.877388    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxh4g\" (UniqueName: \"kubernetes.io/projected/0d0427dc-6080-4116-b38b-90130f37c514-kube-api-access-dxh4g\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:26:05.877602 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.877442    2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d0427dc-6080-4116-b38b-90130f37c514-extensions-socket-volume\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:26:05.978776 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:05.978688    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx"]
Jul 24 12:26:05.983001 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:26:05.982974    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1bfdfe2_e95e_41a5_845b_99b7bc047ce7.slice/crio-758f5302c234046aa467441af42ecb75007039e786c309751a68ae74f0f08d24 WatchSource:0}: Error finding container 758f5302c234046aa467441af42ecb75007039e786c309751a68ae74f0f08d24: Status 404 returned error can't find the container with id 758f5302c234046aa467441af42ecb75007039e786c309751a68ae74f0f08d24
Jul 24 12:26:06.587002 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:06.586959    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx" event={"ID":"e1bfdfe2-e95e-41a5-845b-99b7bc047ce7","Type":"ContainerStarted","Data":"7e02b3ec8287aab06e144f868f971170e676e70e7836e60fe824cd175cf246a2"}
Jul 24 12:26:06.587511 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:06.587030    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx" event={"ID":"e1bfdfe2-e95e-41a5-845b-99b7bc047ce7","Type":"ContainerStarted","Data":"758f5302c234046aa467441af42ecb75007039e786c309751a68ae74f0f08d24"}
Jul 24 12:26:06.587511 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:06.587083    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx"
Jul 24 12:26:06.588355 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:06.588331    2581 scope.go:117] "RemoveContainer" containerID="79eee77756465bab7edfebe237e895d3a350a40afbb2ef71a2782035aed37ead"
Jul 24 12:26:06.588355 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:06.588340    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj"
Jul 24 12:26:06.646475 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:06.646446    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d0427dc-6080-4116-b38b-90130f37c514" path="/var/lib/kubelet/pods/0d0427dc-6080-4116-b38b-90130f37c514/volumes"
Jul 24 12:26:06.665303 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:06.665252    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx" podStartSLOduration=1.6652357 podStartE2EDuration="1.6652357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:26:06.663012339 +0000 UTC m=+398.618017103" watchObservedRunningTime="2026-07-24 12:26:06.6652357 +0000 UTC m=+398.620240462"
Jul 24 12:26:06.668746 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:26:06.668722    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kwfwj\" is forbidden: User \"system:node:ip-10-0-128-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-194.ec2.internal' and this object" podUID="0d0427dc-6080-4116-b38b-90130f37c514" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kwfwj"
Jul 24 12:26:17.597221 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:17.597189    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx"
Jul 24 12:26:32.256109 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.256076    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx"]
Jul 24 12:26:32.256567 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.256278    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx" podUID="e1bfdfe2-e95e-41a5-845b-99b7bc047ce7" containerName="manager" containerID="cri-o://7e02b3ec8287aab06e144f868f971170e676e70e7836e60fe824cd175cf246a2" gracePeriod=10
Jul 24 12:26:32.481579 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.481541    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb"]
Jul 24 12:26:32.483985 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.483969    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb"
Jul 24 12:26:32.501806 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.501786    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx"
Jul 24 12:26:32.508940 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.508883    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb"]
Jul 24 12:26:32.509793 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.509772    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f33a82f6-5582-4e15-9f68-b2478df43388-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mj6xb\" (UID: \"f33a82f6-5582-4e15-9f68-b2478df43388\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb"
Jul 24 12:26:32.509852 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.509822    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn87k\" (UniqueName: \"kubernetes.io/projected/f33a82f6-5582-4e15-9f68-b2478df43388-kube-api-access-pn87k\") pod \"kuadrant-operator-controller-manager-86878c769f-mj6xb\" (UID: \"f33a82f6-5582-4e15-9f68-b2478df43388\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb"
Jul 24 12:26:32.611092 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.611057    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qqk5\" (UniqueName: \"kubernetes.io/projected/e1bfdfe2-e95e-41a5-845b-99b7bc047ce7-kube-api-access-7qqk5\") pod \"e1bfdfe2-e95e-41a5-845b-99b7bc047ce7\" (UID: \"e1bfdfe2-e95e-41a5-845b-99b7bc047ce7\") "
Jul 24 12:26:32.611281 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.611137    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1bfdfe2-e95e-41a5-845b-99b7bc047ce7-extensions-socket-volume\") pod \"e1bfdfe2-e95e-41a5-845b-99b7bc047ce7\" (UID: \"e1bfdfe2-e95e-41a5-845b-99b7bc047ce7\") "
Jul 24 12:26:32.611353 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.611283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f33a82f6-5582-4e15-9f68-b2478df43388-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mj6xb\" (UID: \"f33a82f6-5582-4e15-9f68-b2478df43388\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb"
Jul 24 12:26:32.611353 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.611314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn87k\" (UniqueName: \"kubernetes.io/projected/f33a82f6-5582-4e15-9f68-b2478df43388-kube-api-access-pn87k\") pod \"kuadrant-operator-controller-manager-86878c769f-mj6xb\" (UID: \"f33a82f6-5582-4e15-9f68-b2478df43388\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb"
Jul 24 12:26:32.611548 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.611514    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bfdfe2-e95e-41a5-845b-99b7bc047ce7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e1bfdfe2-e95e-41a5-845b-99b7bc047ce7" (UID: "e1bfdfe2-e95e-41a5-845b-99b7bc047ce7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:26:32.611655 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.611638    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f33a82f6-5582-4e15-9f68-b2478df43388-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mj6xb\" (UID: \"f33a82f6-5582-4e15-9f68-b2478df43388\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb"
Jul 24 12:26:32.613313 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.613288    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1bfdfe2-e95e-41a5-845b-99b7bc047ce7-kube-api-access-7qqk5" (OuterVolumeSpecName: "kube-api-access-7qqk5") pod "e1bfdfe2-e95e-41a5-845b-99b7bc047ce7" (UID: "e1bfdfe2-e95e-41a5-845b-99b7bc047ce7"). InnerVolumeSpecName "kube-api-access-7qqk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:26:32.619876 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.619853    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn87k\" (UniqueName: \"kubernetes.io/projected/f33a82f6-5582-4e15-9f68-b2478df43388-kube-api-access-pn87k\") pod \"kuadrant-operator-controller-manager-86878c769f-mj6xb\" (UID: \"f33a82f6-5582-4e15-9f68-b2478df43388\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb"
Jul 24 12:26:32.713208 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.713178    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qqk5\" (UniqueName: \"kubernetes.io/projected/e1bfdfe2-e95e-41a5-845b-99b7bc047ce7-kube-api-access-7qqk5\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:26:32.713385 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.713221    2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1bfdfe2-e95e-41a5-845b-99b7bc047ce7-extensions-socket-volume\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:26:32.717678 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.717651    2581 generic.go:356] "Generic (PLEG): container finished" podID="e1bfdfe2-e95e-41a5-845b-99b7bc047ce7" containerID="7e02b3ec8287aab06e144f868f971170e676e70e7836e60fe824cd175cf246a2" exitCode=0
Jul 24 12:26:32.717811 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.717691    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx"
Jul 24 12:26:32.717811 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.717727    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx" event={"ID":"e1bfdfe2-e95e-41a5-845b-99b7bc047ce7","Type":"ContainerDied","Data":"7e02b3ec8287aab06e144f868f971170e676e70e7836e60fe824cd175cf246a2"}
Jul 24 12:26:32.717811 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.717779    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx" event={"ID":"e1bfdfe2-e95e-41a5-845b-99b7bc047ce7","Type":"ContainerDied","Data":"758f5302c234046aa467441af42ecb75007039e786c309751a68ae74f0f08d24"}
Jul 24 12:26:32.717811 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.717802    2581 scope.go:117] "RemoveContainer" containerID="7e02b3ec8287aab06e144f868f971170e676e70e7836e60fe824cd175cf246a2"
Jul 24 12:26:32.726789 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.726771    2581 scope.go:117] "RemoveContainer" containerID="7e02b3ec8287aab06e144f868f971170e676e70e7836e60fe824cd175cf246a2"
Jul 24 12:26:32.727056 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:26:32.727042    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e02b3ec8287aab06e144f868f971170e676e70e7836e60fe824cd175cf246a2\": container with ID starting with 7e02b3ec8287aab06e144f868f971170e676e70e7836e60fe824cd175cf246a2 not found: ID does not exist" containerID="7e02b3ec8287aab06e144f868f971170e676e70e7836e60fe824cd175cf246a2"
Jul 24 12:26:32.727122 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.727065    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e02b3ec8287aab06e144f868f971170e676e70e7836e60fe824cd175cf246a2"} err="failed to get container status \"7e02b3ec8287aab06e144f868f971170e676e70e7836e60fe824cd175cf246a2\": rpc error: code = NotFound desc = could not find container \"7e02b3ec8287aab06e144f868f971170e676e70e7836e60fe824cd175cf246a2\": container with ID starting with 7e02b3ec8287aab06e144f868f971170e676e70e7836e60fe824cd175cf246a2 not found: ID does not exist"
Jul 24 12:26:32.739369 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.739342    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx"]
Jul 24 12:26:32.743092 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.743071    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bc9sx"]
Jul 24 12:26:32.808854 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.808816    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb"
Jul 24 12:26:32.940436 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:32.940388    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb"]
Jul 24 12:26:32.943328 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:26:32.943304    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33a82f6_5582_4e15_9f68_b2478df43388.slice/crio-0f96d476913f38c9e1fc4a8bd089b418fbeb7eee8b5315745cfd07d8fad06dd2 WatchSource:0}: Error finding container 0f96d476913f38c9e1fc4a8bd089b418fbeb7eee8b5315745cfd07d8fad06dd2: Status 404 returned error can't find the container with id 0f96d476913f38c9e1fc4a8bd089b418fbeb7eee8b5315745cfd07d8fad06dd2
Jul 24 12:26:33.725167 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:33.725135    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb" event={"ID":"f33a82f6-5582-4e15-9f68-b2478df43388","Type":"ContainerStarted","Data":"7f39a37471f56276e9aad9d88ba20ba85a43edbebd486a0bb0c7660513866c8d"}
Jul 24 12:26:33.725639 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:33.725184    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb" event={"ID":"f33a82f6-5582-4e15-9f68-b2478df43388","Type":"ContainerStarted","Data":"0f96d476913f38c9e1fc4a8bd089b418fbeb7eee8b5315745cfd07d8fad06dd2"}
Jul 24 12:26:33.725639 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:33.725216    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb"
Jul 24 12:26:33.746256 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:33.746204    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb" podStartSLOduration=1.746186882 podStartE2EDuration="1.746186882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:26:33.744014683 +0000 UTC m=+425.699019446" watchObservedRunningTime="2026-07-24 12:26:33.746186882 +0000 UTC m=+425.701191645"
Jul 24 12:26:34.646065 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:34.646022    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1bfdfe2-e95e-41a5-845b-99b7bc047ce7" path="/var/lib/kubelet/pods/e1bfdfe2-e95e-41a5-845b-99b7bc047ce7/volumes"
Jul 24 12:26:44.732799 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:44.732765    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb"
Jul 24 12:26:51.860640 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:51.860557    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:26:51.861022 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:51.860927    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1bfdfe2-e95e-41a5-845b-99b7bc047ce7" containerName="manager"
Jul 24 12:26:51.861022 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:51.860939    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1bfdfe2-e95e-41a5-845b-99b7bc047ce7" containerName="manager"
Jul 24 12:26:51.861022 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:51.860996    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1bfdfe2-e95e-41a5-845b-99b7bc047ce7" containerName="manager"
Jul 24 12:26:51.863865 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:51.863845    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"
Jul 24 12:26:51.866195 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:51.866175    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 24 12:26:51.866270 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:51.866200    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nl2w7\""
Jul 24 12:26:51.877081 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:51.877053    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:26:51.892278 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:51.892232    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:26:51.992076 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:51.992042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jcm6\" (UniqueName: \"kubernetes.io/projected/57515368-5175-44eb-a0f8-841f33ee7427-kube-api-access-2jcm6\") pod \"limitador-limitador-54d6d988d5-w6p8w\" (UID: \"57515368-5175-44eb-a0f8-841f33ee7427\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"
Jul 24 12:26:51.992257 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:51.992133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/57515368-5175-44eb-a0f8-841f33ee7427-config-file\") pod \"limitador-limitador-54d6d988d5-w6p8w\" (UID: \"57515368-5175-44eb-a0f8-841f33ee7427\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"
Jul 24 12:26:52.093429 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:52.093372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jcm6\" (UniqueName: \"kubernetes.io/projected/57515368-5175-44eb-a0f8-841f33ee7427-kube-api-access-2jcm6\") pod \"limitador-limitador-54d6d988d5-w6p8w\" (UID: \"57515368-5175-44eb-a0f8-841f33ee7427\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"
Jul 24 12:26:52.093603 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:52.093517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/57515368-5175-44eb-a0f8-841f33ee7427-config-file\") pod \"limitador-limitador-54d6d988d5-w6p8w\" (UID: \"57515368-5175-44eb-a0f8-841f33ee7427\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"
Jul 24 12:26:52.094161 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:52.094141    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/57515368-5175-44eb-a0f8-841f33ee7427-config-file\") pod \"limitador-limitador-54d6d988d5-w6p8w\" (UID: \"57515368-5175-44eb-a0f8-841f33ee7427\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"
Jul 24 12:26:52.101686 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:52.101658    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jcm6\" (UniqueName: \"kubernetes.io/projected/57515368-5175-44eb-a0f8-841f33ee7427-kube-api-access-2jcm6\") pod \"limitador-limitador-54d6d988d5-w6p8w\" (UID: \"57515368-5175-44eb-a0f8-841f33ee7427\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"
Jul 24 12:26:52.174534 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:52.174449    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"
Jul 24 12:26:52.317011 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:52.316978    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:26:52.320207 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:26:52.320174    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57515368_5175_44eb_a0f8_841f33ee7427.slice/crio-02dfb2a1af90176ec1f9fbdf41ef03dd69839bb01d00130db2ff252514ba30ea WatchSource:0}: Error finding container 02dfb2a1af90176ec1f9fbdf41ef03dd69839bb01d00130db2ff252514ba30ea: Status 404 returned error can't find the container with id 02dfb2a1af90176ec1f9fbdf41ef03dd69839bb01d00130db2ff252514ba30ea
Jul 24 12:26:52.815295 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:52.815216    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-w6p8w" event={"ID":"57515368-5175-44eb-a0f8-841f33ee7427","Type":"ContainerStarted","Data":"02dfb2a1af90176ec1f9fbdf41ef03dd69839bb01d00130db2ff252514ba30ea"}
Jul 24 12:26:55.832572 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:55.832533    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-w6p8w" event={"ID":"57515368-5175-44eb-a0f8-841f33ee7427","Type":"ContainerStarted","Data":"efaebac0b5a689738d7e809a9f1ef52f5d0d2332054e3cec0a943563556ad72e"}
Jul 24 12:26:55.833005 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:55.832656    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"
Jul 24 12:26:55.849128 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:26:55.849075    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-w6p8w" podStartSLOduration=2.28104048 podStartE2EDuration="4.849058527s" totalImagesPullingTime="2.568018047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:26:55.848261172 +0000 UTC m=+447.803265939" watchObservedRunningTime="2026-07-24 12:26:55.849058527 +0000 UTC m=+447.804063534"
Jul 24 12:27:06.838947 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:27:06.838912    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"
Jul 24 12:28:09.296071 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:28:09.296027    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:28:27.098842 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:28:27.098802    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:28:35.204763 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:28:35.204729    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:28:49.496656 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:28:49.496612    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:28:55.805773 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:28:55.805741    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:29:03.398997 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:03.398966    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:29:21.221043 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.221009    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"]
Jul 24 12:29:21.224692 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.224670    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:21.227516 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.227479    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6spvv\""
Jul 24 12:29:21.227637 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.227546    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 24 12:29:21.227637 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.227623    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 24 12:29:21.227801 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.227786    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 24 12:29:21.234220 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.234201    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"]
Jul 24 12:29:21.247937 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.247903    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76911653-0a8a-48b9-ae24-3d3b4fe3b355-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs\" (UID: \"76911653-0a8a-48b9-ae24-3d3b4fe3b355\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:21.248059 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.247995    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76911653-0a8a-48b9-ae24-3d3b4fe3b355-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs\" (UID: \"76911653-0a8a-48b9-ae24-3d3b4fe3b355\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:21.248059 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.248049    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76911653-0a8a-48b9-ae24-3d3b4fe3b355-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs\" (UID: \"76911653-0a8a-48b9-ae24-3d3b4fe3b355\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:21.248178 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.248102    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7jp9\" (UniqueName: \"kubernetes.io/projected/76911653-0a8a-48b9-ae24-3d3b4fe3b355-kube-api-access-h7jp9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs\" (UID: \"76911653-0a8a-48b9-ae24-3d3b4fe3b355\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:21.248178 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.248139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76911653-0a8a-48b9-ae24-3d3b4fe3b355-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs\" (UID: \"76911653-0a8a-48b9-ae24-3d3b4fe3b355\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:21.248178 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.248170    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76911653-0a8a-48b9-ae24-3d3b4fe3b355-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs\" (UID: \"76911653-0a8a-48b9-ae24-3d3b4fe3b355\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:21.349495 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.349451    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76911653-0a8a-48b9-ae24-3d3b4fe3b355-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs\" (UID: \"76911653-0a8a-48b9-ae24-3d3b4fe3b355\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:21.349495 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.349499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76911653-0a8a-48b9-ae24-3d3b4fe3b355-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs\" (UID: \"76911653-0a8a-48b9-ae24-3d3b4fe3b355\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:21.349738 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.349532    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7jp9\" (UniqueName: \"kubernetes.io/projected/76911653-0a8a-48b9-ae24-3d3b4fe3b355-kube-api-access-h7jp9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs\" (UID: \"76911653-0a8a-48b9-ae24-3d3b4fe3b355\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:21.349738 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.349551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76911653-0a8a-48b9-ae24-3d3b4fe3b355-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs\" (UID: \"76911653-0a8a-48b9-ae24-3d3b4fe3b355\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:21.349738 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.349571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76911653-0a8a-48b9-ae24-3d3b4fe3b355-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs\" (UID: \"76911653-0a8a-48b9-ae24-3d3b4fe3b355\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:21.349738 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.349605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76911653-0a8a-48b9-ae24-3d3b4fe3b355-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs\" (UID: \"76911653-0a8a-48b9-ae24-3d3b4fe3b355\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:21.350015 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.349916    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76911653-0a8a-48b9-ae24-3d3b4fe3b355-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs\" (UID: \"76911653-0a8a-48b9-ae24-3d3b4fe3b355\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:21.350015 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.349948    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76911653-0a8a-48b9-ae24-3d3b4fe3b355-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs\" (UID: \"76911653-0a8a-48b9-ae24-3d3b4fe3b355\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:21.350015 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.350001    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76911653-0a8a-48b9-ae24-3d3b4fe3b355-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs\" (UID: \"76911653-0a8a-48b9-ae24-3d3b4fe3b355\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:21.352037 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.352015    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76911653-0a8a-48b9-ae24-3d3b4fe3b355-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs\" (UID: \"76911653-0a8a-48b9-ae24-3d3b4fe3b355\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:21.352128 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.352077    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76911653-0a8a-48b9-ae24-3d3b4fe3b355-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs\" (UID: \"76911653-0a8a-48b9-ae24-3d3b4fe3b355\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:21.358605 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.358583    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7jp9\" (UniqueName: \"kubernetes.io/projected/76911653-0a8a-48b9-ae24-3d3b4fe3b355-kube-api-access-h7jp9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs\" (UID: \"76911653-0a8a-48b9-ae24-3d3b4fe3b355\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:21.534968 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.534925    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:21.678066 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:21.678040    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"]
Jul 24 12:29:21.680460 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:29:21.680430    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76911653_0a8a_48b9_ae24_3d3b4fe3b355.slice/crio-934c5cdb223ead66c79301cc322a5ab2782338f34adebbe7cc1b38619794471a WatchSource:0}: Error finding container 934c5cdb223ead66c79301cc322a5ab2782338f34adebbe7cc1b38619794471a: Status 404 returned error can't find the container with id 934c5cdb223ead66c79301cc322a5ab2782338f34adebbe7cc1b38619794471a
Jul 24 12:29:22.523468 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:22.523428    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs" event={"ID":"76911653-0a8a-48b9-ae24-3d3b4fe3b355","Type":"ContainerStarted","Data":"934c5cdb223ead66c79301cc322a5ab2782338f34adebbe7cc1b38619794471a"}
Jul 24 12:29:23.530318 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:23.530281    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs" event={"ID":"76911653-0a8a-48b9-ae24-3d3b4fe3b355","Type":"ContainerStarted","Data":"41d4409cd1744b52011265b2e0ecf076a38dd5b6cba4983e3712f7f717a10c81"}
Jul 24 12:29:23.550184 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:23.550124    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs" podStartSLOduration=1.534744551 podStartE2EDuration="2.550103433s" totalImagesPullingTime="1.015358882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:29:23.549465043 +0000 UTC m=+595.504469806" watchObservedRunningTime="2026-07-24 12:29:23.550103433 +0000 UTC m=+595.505108195"
Jul 24 12:29:24.005390 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:24.005351    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:29:31.535786 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:31.535742    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:31.536203 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:31.535811    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:31.548519 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:31.548490    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:29:31.579845 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:29:31.579814    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs"
Jul 24 12:30:43.613637 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:30:43.613606    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:30:52.817288 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:30:52.817251    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:31:03.807542 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:31:03.807507    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:32:44.102066 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:32:44.102027    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:32:55.001060 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:32:55.000969    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:33:28.506995 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:33:28.506957    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:33:38.406710 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:33:38.406673    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:33:47.317112 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:33:47.317078    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:33:56.399606 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:33:56.399566    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:34:06.913765 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:34:06.913735    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:34:16.600188 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:34:16.600155    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:34:27.612375 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:34:27.612334    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:34:37.207259 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:34:37.207216    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:34:59.900731 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:34:59.900692    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:35:09.208336 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:35:09.208300    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:35:35.406760 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:35:35.406720    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:35:46.442857 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:35:46.442813    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:36:01.819011 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:36:01.818928    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:36:11.200933 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:36:11.200893    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:36:20.808263 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:36:20.808222    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:36:30.206895 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:36:30.206864    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:36:39.212886 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:36:39.212851    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:37:06.402474 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:37:06.402442    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb"]
Jul 24 12:37:06.403048 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:37:06.402646    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb" podUID="f33a82f6-5582-4e15-9f68-b2478df43388" containerName="manager" containerID="cri-o://7f39a37471f56276e9aad9d88ba20ba85a43edbebd486a0bb0c7660513866c8d" gracePeriod=10
Jul 24 12:37:10.754346 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:37:10.754306    2581 generic.go:356] "Generic (PLEG): container finished" podID="f33a82f6-5582-4e15-9f68-b2478df43388" containerID="7f39a37471f56276e9aad9d88ba20ba85a43edbebd486a0bb0c7660513866c8d" exitCode=0
Jul 24 12:37:10.754723 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:37:10.754380    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb" event={"ID":"f33a82f6-5582-4e15-9f68-b2478df43388","Type":"ContainerDied","Data":"7f39a37471f56276e9aad9d88ba20ba85a43edbebd486a0bb0c7660513866c8d"}
Jul 24 12:37:11.150232 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:37:11.150201    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb"
Jul 24 12:37:11.170367 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:37:11.170339    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn87k\" (UniqueName: \"kubernetes.io/projected/f33a82f6-5582-4e15-9f68-b2478df43388-kube-api-access-pn87k\") pod \"f33a82f6-5582-4e15-9f68-b2478df43388\" (UID: \"f33a82f6-5582-4e15-9f68-b2478df43388\") "
Jul 24 12:37:11.170564 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:37:11.170460    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f33a82f6-5582-4e15-9f68-b2478df43388-extensions-socket-volume\") pod \"f33a82f6-5582-4e15-9f68-b2478df43388\" (UID: \"f33a82f6-5582-4e15-9f68-b2478df43388\") "
Jul 24 12:37:11.170870 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:37:11.170841    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33a82f6-5582-4e15-9f68-b2478df43388-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f33a82f6-5582-4e15-9f68-b2478df43388" (UID: "f33a82f6-5582-4e15-9f68-b2478df43388"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:37:11.173228 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:37:11.173203    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33a82f6-5582-4e15-9f68-b2478df43388-kube-api-access-pn87k" (OuterVolumeSpecName: "kube-api-access-pn87k") pod "f33a82f6-5582-4e15-9f68-b2478df43388" (UID: "f33a82f6-5582-4e15-9f68-b2478df43388"). InnerVolumeSpecName "kube-api-access-pn87k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:37:11.271612 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:37:11.271522    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn87k\" (UniqueName: \"kubernetes.io/projected/f33a82f6-5582-4e15-9f68-b2478df43388-kube-api-access-pn87k\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:37:11.271612 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:37:11.271553    2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f33a82f6-5582-4e15-9f68-b2478df43388-extensions-socket-volume\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:37:11.760135 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:37:11.760097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb" event={"ID":"f33a82f6-5582-4e15-9f68-b2478df43388","Type":"ContainerDied","Data":"0f96d476913f38c9e1fc4a8bd089b418fbeb7eee8b5315745cfd07d8fad06dd2"}
Jul 24 12:37:11.760135 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:37:11.760124    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb"
Jul 24 12:37:11.760753 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:37:11.760162    2581 scope.go:117] "RemoveContainer" containerID="7f39a37471f56276e9aad9d88ba20ba85a43edbebd486a0bb0c7660513866c8d"
Jul 24 12:37:11.782923 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:37:11.782892    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb"]
Jul 24 12:37:11.786712 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:37:11.786692    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mj6xb"]
Jul 24 12:37:12.647072 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:37:12.647038    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f33a82f6-5582-4e15-9f68-b2478df43388" path="/var/lib/kubelet/pods/f33a82f6-5582-4e15-9f68-b2478df43388/volumes"
Jul 24 12:38:07.635120 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:07.635077    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vrkz8"]
Jul 24 12:38:07.635761 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:07.635630    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f33a82f6-5582-4e15-9f68-b2478df43388" containerName="manager"
Jul 24 12:38:07.635761 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:07.635650    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f33a82f6-5582-4e15-9f68-b2478df43388" containerName="manager"
Jul 24 12:38:07.635761 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:07.635758    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f33a82f6-5582-4e15-9f68-b2478df43388" containerName="manager"
Jul 24 12:38:07.638601 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:07.638578    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vrkz8"
Jul 24 12:38:07.642316 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:07.642297    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vgv4q\""
Jul 24 12:38:07.659156 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:07.659129    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vrkz8"]
Jul 24 12:38:07.692088 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:07.692049    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30be1ce8-caf8-45b1-9be2-c50c3ce7f8ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vrkz8\" (UID: \"30be1ce8-caf8-45b1-9be2-c50c3ce7f8ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vrkz8"
Jul 24 12:38:07.692294 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:07.692109    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jps9l\" (UniqueName: \"kubernetes.io/projected/30be1ce8-caf8-45b1-9be2-c50c3ce7f8ea-kube-api-access-jps9l\") pod \"kuadrant-operator-controller-manager-86878c769f-vrkz8\" (UID: \"30be1ce8-caf8-45b1-9be2-c50c3ce7f8ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vrkz8"
Jul 24 12:38:07.792981 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:07.792940    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jps9l\" (UniqueName: \"kubernetes.io/projected/30be1ce8-caf8-45b1-9be2-c50c3ce7f8ea-kube-api-access-jps9l\") pod \"kuadrant-operator-controller-manager-86878c769f-vrkz8\" (UID: \"30be1ce8-caf8-45b1-9be2-c50c3ce7f8ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vrkz8"
Jul 24 12:38:07.793157 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:07.793026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30be1ce8-caf8-45b1-9be2-c50c3ce7f8ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vrkz8\" (UID: \"30be1ce8-caf8-45b1-9be2-c50c3ce7f8ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vrkz8"
Jul 24 12:38:07.793363 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:07.793348    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30be1ce8-caf8-45b1-9be2-c50c3ce7f8ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vrkz8\" (UID: \"30be1ce8-caf8-45b1-9be2-c50c3ce7f8ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vrkz8"
Jul 24 12:38:07.805297 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:07.805273    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jps9l\" (UniqueName: \"kubernetes.io/projected/30be1ce8-caf8-45b1-9be2-c50c3ce7f8ea-kube-api-access-jps9l\") pod \"kuadrant-operator-controller-manager-86878c769f-vrkz8\" (UID: \"30be1ce8-caf8-45b1-9be2-c50c3ce7f8ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vrkz8"
Jul 24 12:38:07.949028 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:07.948939    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vrkz8"
Jul 24 12:38:08.097157 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:08.097134    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vrkz8"]
Jul 24 12:38:08.099401 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:38:08.099374    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30be1ce8_caf8_45b1_9be2_c50c3ce7f8ea.slice/crio-4b2eb69069cb204a96ee1e1b72c026b6f075f817b6490034888371740930e4b0 WatchSource:0}: Error finding container 4b2eb69069cb204a96ee1e1b72c026b6f075f817b6490034888371740930e4b0: Status 404 returned error can't find the container with id 4b2eb69069cb204a96ee1e1b72c026b6f075f817b6490034888371740930e4b0
Jul 24 12:38:09.023241 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:09.023209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vrkz8" event={"ID":"30be1ce8-caf8-45b1-9be2-c50c3ce7f8ea","Type":"ContainerStarted","Data":"98d7f95d3919a7df9bb4dfb458cadbb89520e28259a864a2b273644530faf153"}
Jul 24 12:38:09.023676 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:09.023264    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vrkz8" event={"ID":"30be1ce8-caf8-45b1-9be2-c50c3ce7f8ea","Type":"ContainerStarted","Data":"4b2eb69069cb204a96ee1e1b72c026b6f075f817b6490034888371740930e4b0"}
Jul 24 12:38:09.023676 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:09.023286    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vrkz8"
Jul 24 12:38:09.043771 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:09.043694    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vrkz8" podStartSLOduration=2.043669755 podStartE2EDuration="2.043669755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:38:09.041978882 +0000 UTC m=+1120.996983646" watchObservedRunningTime="2026-07-24 12:38:09.043669755 +0000 UTC m=+1120.998674518"
Jul 24 12:38:20.030176 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:20.030146    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vrkz8"
Jul 24 12:38:27.722058 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:27.722021    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:38:37.008474 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:37.008434    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:38:50.124453 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:38:50.124359    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:39:01.916804 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:39:01.916771    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:39:11.407370 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:39:11.407339    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:39:22.507376 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:39:22.507342    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:39:31.809791 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:39:31.809745    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:39:41.612276 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:39:41.612241    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:39:52.817004 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:39:52.816970    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:40:03.406899 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:40:03.406862    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:40:15.611608 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:40:15.611578    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:40:37.308020 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:40:37.307976    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:40:47.403264 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:40:47.403228    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:41:01.306017 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:01.305986    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:41:08.720916 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.720884    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"]
Jul 24 12:41:08.724460 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.724440    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:08.726637 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.726620    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-e2f4d877-kserve-self-signed-certs\""
Jul 24 12:41:08.733842 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.733822    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"]
Jul 24 12:41:08.874185 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.874152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-dshm\") pod \"e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") " pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:08.874381 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.874238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-home\") pod \"e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") " pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:08.874381 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.874262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-tmp-dir\") pod \"e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") " pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:08.874381 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.874317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-model-cache\") pod \"e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") " pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:08.874381 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.874341    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mlf4\" (UniqueName: \"kubernetes.io/projected/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-kube-api-access-7mlf4\") pod \"e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") " pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:08.874607 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.874399    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-tls-certs\") pod \"e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") " pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:08.974907 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.974813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-model-cache\") pod \"e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") " pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:08.974907 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.974861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mlf4\" (UniqueName: \"kubernetes.io/projected/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-kube-api-access-7mlf4\") pod \"e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") " pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:08.975137 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.974927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-tls-certs\") pod \"e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") " pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:08.975137 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.974984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-dshm\") pod \"e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") " pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:08.975137 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.975040    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-home\") pod \"e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") " pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:08.975137 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.975064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-tmp-dir\") pod \"e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") " pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:08.975298 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.975255    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-model-cache\") pod \"e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") " pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:08.975467 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.975446    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-home\") pod \"e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") " pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:08.975573 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.975499    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-tmp-dir\") pod \"e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") " pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:08.977292 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.977275    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-dshm\") pod \"e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") " pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:08.977654 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.977637    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-tls-certs\") pod \"e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") " pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:08.982898 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:08.982879    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mlf4\" (UniqueName: \"kubernetes.io/projected/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-kube-api-access-7mlf4\") pod \"e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") " pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:09.033964 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:09.033930    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:09.171995 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:09.171947    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"]
Jul 24 12:41:09.174264 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:41:09.174236    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25b15ccb_3113_4c34_aa87_c5bbcb55c9d3.slice/crio-7d72e7f39f54d1fa72f0c2d1af623e355aa3f3a2df805ab3441f77e909653880 WatchSource:0}: Error finding container 7d72e7f39f54d1fa72f0c2d1af623e355aa3f3a2df805ab3441f77e909653880: Status 404 returned error can't find the container with id 7d72e7f39f54d1fa72f0c2d1af623e355aa3f3a2df805ab3441f77e909653880
Jul 24 12:41:09.869284 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:09.869253    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk" event={"ID":"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3","Type":"ContainerStarted","Data":"f766e8a0461573c61905e0da14092d683b3e7bd74e4a85f6ef7885ec62768b21"}
Jul 24 12:41:09.869764 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:09.869296    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk" event={"ID":"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3","Type":"ContainerStarted","Data":"7d72e7f39f54d1fa72f0c2d1af623e355aa3f3a2df805ab3441f77e909653880"}
Jul 24 12:41:09.886908 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:09.886854    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk" podStartSLOduration=1.8868310400000001 podStartE2EDuration="1.88683104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:41:09.886266063 +0000 UTC m=+1301.841270826" watchObservedRunningTime="2026-07-24 12:41:09.88683104 +0000 UTC m=+1301.841835806"
Jul 24 12:41:12.303219 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:12.303190    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:41:19.034219 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:19.034176    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:19.034752 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:19.034247    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:19.046964 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:19.046941    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:19.925956 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:19.925923    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:41:46.157559 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:46.157527    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"]
Jul 24 12:41:49.302964 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:41:49.302884    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:42:01.177448 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:01.177370    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk" podUID="25b15ccb-3113-4c34-aa87-c5bbcb55c9d3" containerName="main" containerID="cri-o://f766e8a0461573c61905e0da14092d683b3e7bd74e4a85f6ef7885ec62768b21" gracePeriod=45
Jul 24 12:42:01.423185 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:01.423157    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:42:01.569087 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:01.569054    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-dshm\") pod \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") "
Jul 24 12:42:01.569087 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:01.569087    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-model-cache\") pod \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") "
Jul 24 12:42:01.569357 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:01.569104    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mlf4\" (UniqueName: \"kubernetes.io/projected/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-kube-api-access-7mlf4\") pod \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") "
Jul 24 12:42:01.569357 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:01.569125    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-tmp-dir\") pod \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") "
Jul 24 12:42:01.569357 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:01.569194    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-tls-certs\") pod \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") "
Jul 24 12:42:01.569357 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:01.569297    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-home\") pod \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\" (UID: \"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3\") "
Jul 24 12:42:01.569603 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:01.569476    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "25b15ccb-3113-4c34-aa87-c5bbcb55c9d3" (UID: "25b15ccb-3113-4c34-aa87-c5bbcb55c9d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:42:01.569603 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:01.569499    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "25b15ccb-3113-4c34-aa87-c5bbcb55c9d3" (UID: "25b15ccb-3113-4c34-aa87-c5bbcb55c9d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:42:01.569705 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:01.569642    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-model-cache\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:42:01.569705 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:01.569656    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-home" (OuterVolumeSpecName: "home") pod "25b15ccb-3113-4c34-aa87-c5bbcb55c9d3" (UID: "25b15ccb-3113-4c34-aa87-c5bbcb55c9d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:42:01.569705 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:01.569685    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-tmp-dir\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:42:01.571715 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:01.571682    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-dshm" (OuterVolumeSpecName: "dshm") pod "25b15ccb-3113-4c34-aa87-c5bbcb55c9d3" (UID: "25b15ccb-3113-4c34-aa87-c5bbcb55c9d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:42:01.571848 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:01.571719    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-kube-api-access-7mlf4" (OuterVolumeSpecName: "kube-api-access-7mlf4") pod "25b15ccb-3113-4c34-aa87-c5bbcb55c9d3" (UID: "25b15ccb-3113-4c34-aa87-c5bbcb55c9d3"). InnerVolumeSpecName "kube-api-access-7mlf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:42:01.571848 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:01.571752    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25b15ccb-3113-4c34-aa87-c5bbcb55c9d3" (UID: "25b15ccb-3113-4c34-aa87-c5bbcb55c9d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:42:01.671031 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:01.670997    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-tls-certs\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:42:01.671031 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:01.671024    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-home\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:42:01.671031 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:01.671032    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-dshm\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:42:01.671031 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:01.671040    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mlf4\" (UniqueName: \"kubernetes.io/projected/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3-kube-api-access-7mlf4\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:42:02.122693 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:02.122662    2581 generic.go:356] "Generic (PLEG): container finished" podID="25b15ccb-3113-4c34-aa87-c5bbcb55c9d3" containerID="f766e8a0461573c61905e0da14092d683b3e7bd74e4a85f6ef7885ec62768b21" exitCode=0
Jul 24 12:42:02.122879 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:02.122710    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"
Jul 24 12:42:02.122879 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:02.122713    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk" event={"ID":"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3","Type":"ContainerDied","Data":"f766e8a0461573c61905e0da14092d683b3e7bd74e4a85f6ef7885ec62768b21"}
Jul 24 12:42:02.122879 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:02.122817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk" event={"ID":"25b15ccb-3113-4c34-aa87-c5bbcb55c9d3","Type":"ContainerDied","Data":"7d72e7f39f54d1fa72f0c2d1af623e355aa3f3a2df805ab3441f77e909653880"}
Jul 24 12:42:02.122879 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:02.122834    2581 scope.go:117] "RemoveContainer" containerID="f766e8a0461573c61905e0da14092d683b3e7bd74e4a85f6ef7885ec62768b21"
Jul 24 12:42:02.131823 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:02.131805    2581 scope.go:117] "RemoveContainer" containerID="f766e8a0461573c61905e0da14092d683b3e7bd74e4a85f6ef7885ec62768b21"
Jul 24 12:42:02.132079 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:42:02.132062    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f766e8a0461573c61905e0da14092d683b3e7bd74e4a85f6ef7885ec62768b21\": container with ID starting with f766e8a0461573c61905e0da14092d683b3e7bd74e4a85f6ef7885ec62768b21 not found: ID does not exist" containerID="f766e8a0461573c61905e0da14092d683b3e7bd74e4a85f6ef7885ec62768b21"
Jul 24 12:42:02.132130 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:02.132088    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f766e8a0461573c61905e0da14092d683b3e7bd74e4a85f6ef7885ec62768b21"} err="failed to get container status \"f766e8a0461573c61905e0da14092d683b3e7bd74e4a85f6ef7885ec62768b21\": rpc error: code = NotFound desc = could not find container \"f766e8a0461573c61905e0da14092d683b3e7bd74e4a85f6ef7885ec62768b21\": container with ID starting with f766e8a0461573c61905e0da14092d683b3e7bd74e4a85f6ef7885ec62768b21 not found: ID does not exist"
Jul 24 12:42:02.146388 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:02.146355    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"]
Jul 24 12:42:02.149184 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:02.149163    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-e2f4d877-kserve-775846c595-nqtxk"]
Jul 24 12:42:02.647406 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:02.647368    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25b15ccb-3113-4c34-aa87-c5bbcb55c9d3" path="/var/lib/kubelet/pods/25b15ccb-3113-4c34-aa87-c5bbcb55c9d3/volumes"
Jul 24 12:42:03.413177 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:03.413136    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:42:13.425390 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:13.425352    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:42:27.446609 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:27.446574    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:42:37.149045 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:37.149006    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:42:46.513350 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:46.513313    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:42:56.708711 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:42:56.708676    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:43:05.911556 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:43:05.911522    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:43:15.315117 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:43:15.315081    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:43:28.397511 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:43:28.397469    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:43:37.510147 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:43:37.510101    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:43:49.119211 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:43:49.119175    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:44:01.309490 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:44:01.309458    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:44:14.300723 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:44:14.300688    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:44:22.103688 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:44:22.103652    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:44:33.011558 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:44:33.011518    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:44:44.607549 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:44:44.607508    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:44:57.697658 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:44:57.697578    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:45:10.099615 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:45:10.099574    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:45:19.701047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:45:19.701005    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:45:29.599838 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:45:29.599805    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:45:39.797537 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:45:39.797502    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:45:56.099552 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:45:56.099517    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:46:07.001878 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:07.001847    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:46:26.945579 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:26.945545    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0a74db1-f7mgq"]
Jul 24 12:46:26.946073 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:26.945899    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25b15ccb-3113-4c34-aa87-c5bbcb55c9d3" containerName="main"
Jul 24 12:46:26.946073 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:26.945910    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="25b15ccb-3113-4c34-aa87-c5bbcb55c9d3" containerName="main"
Jul 24 12:46:26.946073 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:26.945980    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="25b15ccb-3113-4c34-aa87-c5bbcb55c9d3" containerName="main"
Jul 24 12:46:26.949132 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:26.949109    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0a74db1-f7mgq"
Jul 24 12:46:26.951291 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:26.951268    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 12:46:26.951291 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:26.951286    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zw8hg\""
Jul 24 12:46:26.956952 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:26.956498    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0a74db1-f7mgq"]
Jul 24 12:46:27.063997 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:27.063956    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce36ba7b-a0fc-4129-9050-1b8269ee5ed3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b0a74db1-f7mgq\" (UID: \"ce36ba7b-a0fc-4129-9050-1b8269ee5ed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0a74db1-f7mgq"
Jul 24 12:46:27.165356 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:27.165314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce36ba7b-a0fc-4129-9050-1b8269ee5ed3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b0a74db1-f7mgq\" (UID: \"ce36ba7b-a0fc-4129-9050-1b8269ee5ed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0a74db1-f7mgq"
Jul 24 12:46:27.166064 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:27.166037    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce36ba7b-a0fc-4129-9050-1b8269ee5ed3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b0a74db1-f7mgq\" (UID: \"ce36ba7b-a0fc-4129-9050-1b8269ee5ed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0a74db1-f7mgq"
Jul 24 12:46:27.260648 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:27.260540    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0a74db1-f7mgq"
Jul 24 12:46:27.396727 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:27.396705    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0a74db1-f7mgq"]
Jul 24 12:46:27.399228 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:46:27.399202    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce36ba7b_a0fc_4129_9050_1b8269ee5ed3.slice/crio-200205c3c1027b32f8f5ed9238a8c03a0fd13793bef3e7409006c44ee542e00d WatchSource:0}: Error finding container 200205c3c1027b32f8f5ed9238a8c03a0fd13793bef3e7409006c44ee542e00d: Status 404 returned error can't find the container with id 200205c3c1027b32f8f5ed9238a8c03a0fd13793bef3e7409006c44ee542e00d
Jul 24 12:46:27.400951 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:27.400934    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 12:46:28.399476 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:28.399438    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0a74db1-f7mgq" event={"ID":"ce36ba7b-a0fc-4129-9050-1b8269ee5ed3","Type":"ContainerStarted","Data":"200205c3c1027b32f8f5ed9238a8c03a0fd13793bef3e7409006c44ee542e00d"}
Jul 24 12:46:30.410503 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:30.410463    2581 generic.go:356] "Generic (PLEG): container finished" podID="ce36ba7b-a0fc-4129-9050-1b8269ee5ed3" containerID="17d4f5a539bf77a9ccee765ea641292124300a60b98237bdfec2e2e47a4b2847" exitCode=0
Jul 24 12:46:30.410863 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:30.410551    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0a74db1-f7mgq" event={"ID":"ce36ba7b-a0fc-4129-9050-1b8269ee5ed3","Type":"ContainerDied","Data":"17d4f5a539bf77a9ccee765ea641292124300a60b98237bdfec2e2e47a4b2847"}
Jul 24 12:46:31.553636 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:31.553611    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0a74db1-f7mgq"
Jul 24 12:46:31.714504 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:31.714392    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce36ba7b-a0fc-4129-9050-1b8269ee5ed3-maas-api-service-ca\") pod \"ce36ba7b-a0fc-4129-9050-1b8269ee5ed3\" (UID: \"ce36ba7b-a0fc-4129-9050-1b8269ee5ed3\") "
Jul 24 12:46:31.714789 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:31.714765    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce36ba7b-a0fc-4129-9050-1b8269ee5ed3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ce36ba7b-a0fc-4129-9050-1b8269ee5ed3" (UID: "ce36ba7b-a0fc-4129-9050-1b8269ee5ed3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:46:31.815909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:31.815860    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce36ba7b-a0fc-4129-9050-1b8269ee5ed3-maas-api-service-ca\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:46:32.420759 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:32.420730    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0a74db1-f7mgq"
Jul 24 12:46:32.421065 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:32.420728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0a74db1-f7mgq" event={"ID":"ce36ba7b-a0fc-4129-9050-1b8269ee5ed3","Type":"ContainerDied","Data":"200205c3c1027b32f8f5ed9238a8c03a0fd13793bef3e7409006c44ee542e00d"}
Jul 24 12:46:32.421065 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:32.420856    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="200205c3c1027b32f8f5ed9238a8c03a0fd13793bef3e7409006c44ee542e00d"
Jul 24 12:46:40.496736 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:40.496696    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:46:42.045365 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:42.045313    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0a74db1-f7mgq"]
Jul 24 12:46:42.047158 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:42.047135    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0a74db1-f7mgq"]
Jul 24 12:46:42.645153 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:42.645121    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce36ba7b-a0fc-4129-9050-1b8269ee5ed3" path="/var/lib/kubelet/pods/ce36ba7b-a0fc-4129-9050-1b8269ee5ed3/volumes"
Jul 24 12:46:51.513068 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:46:51.513028    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:47:11.176327 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:11.176291    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52688321-rjrvh"]
Jul 24 12:47:11.176812 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:11.176720    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce36ba7b-a0fc-4129-9050-1b8269ee5ed3" containerName="revoke-keys"
Jul 24 12:47:11.176812 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:11.176734    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce36ba7b-a0fc-4129-9050-1b8269ee5ed3" containerName="revoke-keys"
Jul 24 12:47:11.176812 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:11.176788    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce36ba7b-a0fc-4129-9050-1b8269ee5ed3" containerName="revoke-keys"
Jul 24 12:47:11.184261 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:11.184238    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52688321-rjrvh"
Jul 24 12:47:11.186955 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:11.186928    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zw8hg\""
Jul 24 12:47:11.187151 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:11.186934    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 12:47:11.187411 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:11.187384    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52688321-rjrvh"]
Jul 24 12:47:11.282707 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:11.282672    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e6d8463-191f-430c-b130-946daf8b28f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52688321-rjrvh\" (UID: \"8e6d8463-191f-430c-b130-946daf8b28f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52688321-rjrvh"
Jul 24 12:47:11.383650 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:11.383612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e6d8463-191f-430c-b130-946daf8b28f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52688321-rjrvh\" (UID: \"8e6d8463-191f-430c-b130-946daf8b28f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52688321-rjrvh"
Jul 24 12:47:11.384234 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:11.384214    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e6d8463-191f-430c-b130-946daf8b28f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52688321-rjrvh\" (UID: \"8e6d8463-191f-430c-b130-946daf8b28f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52688321-rjrvh"
Jul 24 12:47:11.496657 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:11.496566    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52688321-rjrvh"
Jul 24 12:47:11.630359 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:11.630315    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52688321-rjrvh"]
Jul 24 12:47:11.633563 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:47:11.633536    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6d8463_191f_430c_b130_946daf8b28f8.slice/crio-6e5164a68e9b1535f8736c577fd74b8edd60126fb1269af5abeccb3dcb15ebb5 WatchSource:0}: Error finding container 6e5164a68e9b1535f8736c577fd74b8edd60126fb1269af5abeccb3dcb15ebb5: Status 404 returned error can't find the container with id 6e5164a68e9b1535f8736c577fd74b8edd60126fb1269af5abeccb3dcb15ebb5
Jul 24 12:47:12.610247 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:12.610209    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e6d8463-191f-430c-b130-946daf8b28f8" containerID="f8e15cac94a193660251fc4cdad579b33c06ce79d14bd40aad5852adb516f821" exitCode=0
Jul 24 12:47:12.610655 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:12.610294    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52688321-rjrvh" event={"ID":"8e6d8463-191f-430c-b130-946daf8b28f8","Type":"ContainerDied","Data":"f8e15cac94a193660251fc4cdad579b33c06ce79d14bd40aad5852adb516f821"}
Jul 24 12:47:12.610655 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:12.610376    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52688321-rjrvh" event={"ID":"8e6d8463-191f-430c-b130-946daf8b28f8","Type":"ContainerStarted","Data":"6e5164a68e9b1535f8736c577fd74b8edd60126fb1269af5abeccb3dcb15ebb5"}
Jul 24 12:47:13.744898 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:13.744875    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52688321-rjrvh"
Jul 24 12:47:13.804382 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:13.804352    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e6d8463-191f-430c-b130-946daf8b28f8-maas-api-service-ca\") pod \"8e6d8463-191f-430c-b130-946daf8b28f8\" (UID: \"8e6d8463-191f-430c-b130-946daf8b28f8\") "
Jul 24 12:47:13.804751 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:13.804724    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6d8463-191f-430c-b130-946daf8b28f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8e6d8463-191f-430c-b130-946daf8b28f8" (UID: "8e6d8463-191f-430c-b130-946daf8b28f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:47:13.905547 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:13.905456    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e6d8463-191f-430c-b130-946daf8b28f8-maas-api-service-ca\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:47:14.620891 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:14.620860    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52688321-rjrvh"
Jul 24 12:47:14.620891 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:14.620882    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52688321-rjrvh" event={"ID":"8e6d8463-191f-430c-b130-946daf8b28f8","Type":"ContainerDied","Data":"6e5164a68e9b1535f8736c577fd74b8edd60126fb1269af5abeccb3dcb15ebb5"}
Jul 24 12:47:14.621146 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:14.620925    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e5164a68e9b1535f8736c577fd74b8edd60126fb1269af5abeccb3dcb15ebb5"
Jul 24 12:47:25.411388 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:25.411351    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:47:26.279872 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:26.279837    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52688321-rjrvh"]
Jul 24 12:47:26.281347 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:26.281325    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52688321-rjrvh"]
Jul 24 12:47:26.645345 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:26.645310    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e6d8463-191f-430c-b130-946daf8b28f8" path="/var/lib/kubelet/pods/8e6d8463-191f-430c-b130-946daf8b28f8/volumes"
Jul 24 12:47:36.499335 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:36.499299    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:47:56.165741 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:56.165660    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-266bd0de-hktxs"]
Jul 24 12:47:56.166186 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:56.166067    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e6d8463-191f-430c-b130-946daf8b28f8" containerName="revoke-keys"
Jul 24 12:47:56.166186 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:56.166078    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e6d8463-191f-430c-b130-946daf8b28f8" containerName="revoke-keys"
Jul 24 12:47:56.166186 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:56.166137    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e6d8463-191f-430c-b130-946daf8b28f8" containerName="revoke-keys"
Jul 24 12:47:56.169458 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:56.169440    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-266bd0de-hktxs"
Jul 24 12:47:56.171728 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:56.171705    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zw8hg\""
Jul 24 12:47:56.171855 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:56.171769    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 12:47:56.177685 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:56.177623    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-266bd0de-hktxs"]
Jul 24 12:47:56.197766 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:56.197735    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/857f1a16-3358-46a1-8daf-09a1c9e78e76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-266bd0de-hktxs\" (UID: \"857f1a16-3358-46a1-8daf-09a1c9e78e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-266bd0de-hktxs"
Jul 24 12:47:56.298514 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:56.298474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/857f1a16-3358-46a1-8daf-09a1c9e78e76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-266bd0de-hktxs\" (UID: \"857f1a16-3358-46a1-8daf-09a1c9e78e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-266bd0de-hktxs"
Jul 24 12:47:56.299139 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:56.299119    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/857f1a16-3358-46a1-8daf-09a1c9e78e76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-266bd0de-hktxs\" (UID: \"857f1a16-3358-46a1-8daf-09a1c9e78e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-266bd0de-hktxs"
Jul 24 12:47:56.481052 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:56.480955    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-266bd0de-hktxs"
Jul 24 12:47:56.612102 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:56.612080    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-266bd0de-hktxs"]
Jul 24 12:47:56.613957 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:47:56.613930    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857f1a16_3358_46a1_8daf_09a1c9e78e76.slice/crio-1558e78a49c87939499544af60973201cbe3d952d971ae7eeba49b82f795fa17 WatchSource:0}: Error finding container 1558e78a49c87939499544af60973201cbe3d952d971ae7eeba49b82f795fa17: Status 404 returned error can't find the container with id 1558e78a49c87939499544af60973201cbe3d952d971ae7eeba49b82f795fa17
Jul 24 12:47:56.826218 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:56.826180    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-266bd0de-hktxs" event={"ID":"857f1a16-3358-46a1-8daf-09a1c9e78e76","Type":"ContainerStarted","Data":"1558e78a49c87939499544af60973201cbe3d952d971ae7eeba49b82f795fa17"}
Jul 24 12:47:57.832585 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:57.832555    2581 generic.go:356] "Generic (PLEG): container finished" podID="857f1a16-3358-46a1-8daf-09a1c9e78e76" containerID="7b999701e7bc51bd9b8181fb75a842866a98908d954a6d21b620b9a0677c32ce" exitCode=0
Jul 24 12:47:57.832985 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:57.832635    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-266bd0de-hktxs" event={"ID":"857f1a16-3358-46a1-8daf-09a1c9e78e76","Type":"ContainerDied","Data":"7b999701e7bc51bd9b8181fb75a842866a98908d954a6d21b620b9a0677c32ce"}
Jul 24 12:47:58.966166 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:58.966145    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-266bd0de-hktxs"
Jul 24 12:47:59.020962 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:59.020935    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/857f1a16-3358-46a1-8daf-09a1c9e78e76-maas-api-service-ca\") pod \"857f1a16-3358-46a1-8daf-09a1c9e78e76\" (UID: \"857f1a16-3358-46a1-8daf-09a1c9e78e76\") "
Jul 24 12:47:59.021303 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:59.021283    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/857f1a16-3358-46a1-8daf-09a1c9e78e76-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "857f1a16-3358-46a1-8daf-09a1c9e78e76" (UID: "857f1a16-3358-46a1-8daf-09a1c9e78e76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:47:59.122439 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:59.122325    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/857f1a16-3358-46a1-8daf-09a1c9e78e76-maas-api-service-ca\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:47:59.842460 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:59.842436    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-266bd0de-hktxs"
Jul 24 12:47:59.842636 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:59.842493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-266bd0de-hktxs" event={"ID":"857f1a16-3358-46a1-8daf-09a1c9e78e76","Type":"ContainerDied","Data":"1558e78a49c87939499544af60973201cbe3d952d971ae7eeba49b82f795fa17"}
Jul 24 12:47:59.842636 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:47:59.842548    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1558e78a49c87939499544af60973201cbe3d952d971ae7eeba49b82f795fa17"
Jul 24 12:48:10.100833 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:48:10.100798    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:48:11.256475 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:48:11.256443    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-266bd0de-hktxs"]
Jul 24 12:48:11.258449 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:48:11.258405    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-266bd0de-hktxs"]
Jul 24 12:48:12.646514 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:48:12.646482    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="857f1a16-3358-46a1-8daf-09a1c9e78e76" path="/var/lib/kubelet/pods/857f1a16-3358-46a1-8daf-09a1c9e78e76/volumes"
Jul 24 12:49:37.901193 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:49:37.901155    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:49:50.213871 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:49:50.213837    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:50:00.810522 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:00.810485    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:50:08.108555 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:08.108519    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e38b46e9-v2m6r"]
Jul 24 12:50:08.109034 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:08.108953    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="857f1a16-3358-46a1-8daf-09a1c9e78e76" containerName="revoke-keys"
Jul 24 12:50:08.109034 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:08.108965    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="857f1a16-3358-46a1-8daf-09a1c9e78e76" containerName="revoke-keys"
Jul 24 12:50:08.109034 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:08.109022    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="857f1a16-3358-46a1-8daf-09a1c9e78e76" containerName="revoke-keys"
Jul 24 12:50:08.112122 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:08.112106    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e38b46e9-v2m6r"
Jul 24 12:50:08.114227 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:08.114202    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 12:50:08.114344 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:08.114276    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zw8hg\""
Jul 24 12:50:08.119749 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:08.119602    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e38b46e9-v2m6r"]
Jul 24 12:50:08.208022 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:08.207982    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/306a330b-de24-4452-978c-dc38616c4e0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e38b46e9-v2m6r\" (UID: \"306a330b-de24-4452-978c-dc38616c4e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e38b46e9-v2m6r"
Jul 24 12:50:08.308843 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:08.308801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/306a330b-de24-4452-978c-dc38616c4e0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e38b46e9-v2m6r\" (UID: \"306a330b-de24-4452-978c-dc38616c4e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e38b46e9-v2m6r"
Jul 24 12:50:08.309479 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:08.309459    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/306a330b-de24-4452-978c-dc38616c4e0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e38b46e9-v2m6r\" (UID: \"306a330b-de24-4452-978c-dc38616c4e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e38b46e9-v2m6r"
Jul 24 12:50:08.423407 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:08.423325    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e38b46e9-v2m6r"
Jul 24 12:50:08.553723 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:08.553665    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e38b46e9-v2m6r"]
Jul 24 12:50:08.556628 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:50:08.556602    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306a330b_de24_4452_978c_dc38616c4e0b.slice/crio-d6a108efc864da409b6cc2bd13783700c5d5f9f937dfbf505aa432a9c9260412 WatchSource:0}: Error finding container d6a108efc864da409b6cc2bd13783700c5d5f9f937dfbf505aa432a9c9260412: Status 404 returned error can't find the container with id d6a108efc864da409b6cc2bd13783700c5d5f9f937dfbf505aa432a9c9260412
Jul 24 12:50:09.459111 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:09.459072    2581 generic.go:356] "Generic (PLEG): container finished" podID="306a330b-de24-4452-978c-dc38616c4e0b" containerID="0191d8d788c1b26a0a5600b06d50ea9132b044bfc1918efef33d76683557788e" exitCode=0
Jul 24 12:50:09.459556 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:09.459123    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e38b46e9-v2m6r" event={"ID":"306a330b-de24-4452-978c-dc38616c4e0b","Type":"ContainerDied","Data":"0191d8d788c1b26a0a5600b06d50ea9132b044bfc1918efef33d76683557788e"}
Jul 24 12:50:09.459556 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:09.459192    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e38b46e9-v2m6r" event={"ID":"306a330b-de24-4452-978c-dc38616c4e0b","Type":"ContainerStarted","Data":"d6a108efc864da409b6cc2bd13783700c5d5f9f937dfbf505aa432a9c9260412"}
Jul 24 12:50:10.594522 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:10.594499    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e38b46e9-v2m6r"
Jul 24 12:50:10.730013 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:10.729924    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/306a330b-de24-4452-978c-dc38616c4e0b-maas-api-service-ca\") pod \"306a330b-de24-4452-978c-dc38616c4e0b\" (UID: \"306a330b-de24-4452-978c-dc38616c4e0b\") "
Jul 24 12:50:10.730299 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:10.730274    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306a330b-de24-4452-978c-dc38616c4e0b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "306a330b-de24-4452-978c-dc38616c4e0b" (UID: "306a330b-de24-4452-978c-dc38616c4e0b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:50:10.830634 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:10.830602    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/306a330b-de24-4452-978c-dc38616c4e0b-maas-api-service-ca\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:50:11.470593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:11.470560    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e38b46e9-v2m6r" event={"ID":"306a330b-de24-4452-978c-dc38616c4e0b","Type":"ContainerDied","Data":"d6a108efc864da409b6cc2bd13783700c5d5f9f937dfbf505aa432a9c9260412"}
Jul 24 12:50:11.470593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:11.470594    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e38b46e9-v2m6r"
Jul 24 12:50:11.470822 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:11.470609    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6a108efc864da409b6cc2bd13783700c5d5f9f937dfbf505aa432a9c9260412"
Jul 24 12:50:21.601410 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:21.601373    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:50:23.188875 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:23.188844    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e38b46e9-v2m6r"]
Jul 24 12:50:23.191489 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:23.191465    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e38b46e9-v2m6r"]
Jul 24 12:50:24.653009 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:24.652971    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="306a330b-de24-4452-978c-dc38616c4e0b" path="/var/lib/kubelet/pods/306a330b-de24-4452-978c-dc38616c4e0b/volumes"
Jul 24 12:50:58.799979 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:50:58.799895    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:51:07.505846 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:07.505808    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:51:22.102843 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:22.102806    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:51:26.863494 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:26.863455    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f55af331-zq5sk"]
Jul 24 12:51:26.863877 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:26.863832    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="306a330b-de24-4452-978c-dc38616c4e0b" containerName="revoke-keys"
Jul 24 12:51:26.863877 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:26.863843    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="306a330b-de24-4452-978c-dc38616c4e0b" containerName="revoke-keys"
Jul 24 12:51:26.863948 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:26.863903    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="306a330b-de24-4452-978c-dc38616c4e0b" containerName="revoke-keys"
Jul 24 12:51:26.867104 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:26.867087    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f55af331-zq5sk"
Jul 24 12:51:26.869060 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:26.869043    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 12:51:26.869155 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:26.869102    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zw8hg\""
Jul 24 12:51:26.873327 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:26.873300    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f55af331-zq5sk"]
Jul 24 12:51:27.027696 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:27.027655    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a96a7b53-b3db-4298-a41b-b45d183d683b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f55af331-zq5sk\" (UID: \"a96a7b53-b3db-4298-a41b-b45d183d683b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f55af331-zq5sk"
Jul 24 12:51:27.128981 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:27.128893    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a96a7b53-b3db-4298-a41b-b45d183d683b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f55af331-zq5sk\" (UID: \"a96a7b53-b3db-4298-a41b-b45d183d683b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f55af331-zq5sk"
Jul 24 12:51:27.129616 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:27.129593    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a96a7b53-b3db-4298-a41b-b45d183d683b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f55af331-zq5sk\" (UID: \"a96a7b53-b3db-4298-a41b-b45d183d683b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f55af331-zq5sk"
Jul 24 12:51:27.179395 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:27.179354    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f55af331-zq5sk"
Jul 24 12:51:27.311513 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:27.311485    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f55af331-zq5sk"]
Jul 24 12:51:27.313122 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:51:27.313092    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96a7b53_b3db_4298_a41b_b45d183d683b.slice/crio-b783e515e4efd452d62a0aa95ad159d5997da85c509185fe4b99c6be556c2c2b WatchSource:0}: Error finding container b783e515e4efd452d62a0aa95ad159d5997da85c509185fe4b99c6be556c2c2b: Status 404 returned error can't find the container with id b783e515e4efd452d62a0aa95ad159d5997da85c509185fe4b99c6be556c2c2b
Jul 24 12:51:27.843488 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:27.843376    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f55af331-zq5sk" event={"ID":"a96a7b53-b3db-4298-a41b-b45d183d683b","Type":"ContainerStarted","Data":"f215a4a68b730f31583f84326cb1407d9614e4dfece52759e9b5259236697472"}
Jul 24 12:51:27.843488 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:27.843456    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f55af331-zq5sk" event={"ID":"a96a7b53-b3db-4298-a41b-b45d183d683b","Type":"ContainerStarted","Data":"b783e515e4efd452d62a0aa95ad159d5997da85c509185fe4b99c6be556c2c2b"}
Jul 24 12:51:27.859672 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:27.859607    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f55af331-zq5sk" podStartSLOduration=1.859591409 podStartE2EDuration="1.859591409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:51:27.857517209 +0000 UTC m=+1919.812522006" watchObservedRunningTime="2026-07-24 12:51:27.859591409 +0000 UTC m=+1919.814596150"
Jul 24 12:51:28.850626 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:28.850590    2581 generic.go:356] "Generic (PLEG): container finished" podID="a96a7b53-b3db-4298-a41b-b45d183d683b" containerID="f215a4a68b730f31583f84326cb1407d9614e4dfece52759e9b5259236697472" exitCode=0
Jul 24 12:51:28.851055 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:28.850674    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f55af331-zq5sk" event={"ID":"a96a7b53-b3db-4298-a41b-b45d183d683b","Type":"ContainerDied","Data":"f215a4a68b730f31583f84326cb1407d9614e4dfece52759e9b5259236697472"}
Jul 24 12:51:29.987629 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:29.987607    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f55af331-zq5sk"
Jul 24 12:51:30.159076 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:30.158971    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a96a7b53-b3db-4298-a41b-b45d183d683b-maas-api-service-ca\") pod \"a96a7b53-b3db-4298-a41b-b45d183d683b\" (UID: \"a96a7b53-b3db-4298-a41b-b45d183d683b\") "
Jul 24 12:51:30.159356 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:30.159332    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a96a7b53-b3db-4298-a41b-b45d183d683b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a96a7b53-b3db-4298-a41b-b45d183d683b" (UID: "a96a7b53-b3db-4298-a41b-b45d183d683b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:51:30.259747 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:30.259704    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a96a7b53-b3db-4298-a41b-b45d183d683b-maas-api-service-ca\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:51:30.862487 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:30.862457    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f55af331-zq5sk"
Jul 24 12:51:30.862652 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:30.862459    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f55af331-zq5sk" event={"ID":"a96a7b53-b3db-4298-a41b-b45d183d683b","Type":"ContainerDied","Data":"b783e515e4efd452d62a0aa95ad159d5997da85c509185fe4b99c6be556c2c2b"}
Jul 24 12:51:30.862652 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:30.862575    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b783e515e4efd452d62a0aa95ad159d5997da85c509185fe4b99c6be556c2c2b"
Jul 24 12:51:41.201567 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:41.201534    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:51:46.957343 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:46.957206    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f55af331-zq5sk"]
Jul 24 12:51:46.959017 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:46.958998    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f55af331-zq5sk"]
Jul 24 12:51:48.646763 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:48.646728    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a96a7b53-b3db-4298-a41b-b45d183d683b" path="/var/lib/kubelet/pods/a96a7b53-b3db-4298-a41b-b45d183d683b/volumes"
Jul 24 12:51:50.102892 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:51:50.102856    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:52:42.902747 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:52:42.902711    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:52:52.601519 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:52:52.601483    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:53:28.710386 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:53:28.710356    2581 scope.go:117] "RemoveContainer" containerID="f8e15cac94a193660251fc4cdad579b33c06ce79d14bd40aad5852adb516f821"
Jul 24 12:53:28.718447 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:53:28.718404    2581 scope.go:117] "RemoveContainer" containerID="17d4f5a539bf77a9ccee765ea641292124300a60b98237bdfec2e2e47a4b2847"
Jul 24 12:53:43.521030 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:53:43.520986    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:53:55.697476 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:53:55.697361    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:54:03.820952 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.820918    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"]
Jul 24 12:54:03.821453 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.821283    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a96a7b53-b3db-4298-a41b-b45d183d683b" containerName="revoke-keys"
Jul 24 12:54:03.821453 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.821293    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a96a7b53-b3db-4298-a41b-b45d183d683b" containerName="revoke-keys"
Jul 24 12:54:03.821453 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.821352    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a96a7b53-b3db-4298-a41b-b45d183d683b" containerName="revoke-keys"
Jul 24 12:54:03.824239 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.824223    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:03.827041 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.827020    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-dd4baf\"/\"test-model-dd4baf-kserve-self-signed-certs\""
Jul 24 12:54:03.827153 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.827054    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-dd4baf\"/\"default-dockercfg-rvmvn\""
Jul 24 12:54:03.827153 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.827061    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-dd4baf\"/\"kube-root-ca.crt\""
Jul 24 12:54:03.827267 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.827155    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-dd4baf\"/\"openshift-service-ca.crt\""
Jul 24 12:54:03.833276 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.833254    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"]
Jul 24 12:54:03.873496 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.873464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-tmp-dir\") pod \"test-model-dd4baf-kserve-66f55ccfc9-cprfk\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") " pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:03.873682 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.873549    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-model-cache\") pod \"test-model-dd4baf-kserve-66f55ccfc9-cprfk\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") " pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:03.873682 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.873628    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-home\") pod \"test-model-dd4baf-kserve-66f55ccfc9-cprfk\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") " pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:03.873682 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.873653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-dshm\") pod \"test-model-dd4baf-kserve-66f55ccfc9-cprfk\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") " pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:03.873682 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.873678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68547b35-835e-483c-9f0b-0169e1ea1d74-tls-certs\") pod \"test-model-dd4baf-kserve-66f55ccfc9-cprfk\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") " pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:03.873888 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.873711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kfzx\" (UniqueName: \"kubernetes.io/projected/68547b35-835e-483c-9f0b-0169e1ea1d74-kube-api-access-5kfzx\") pod \"test-model-dd4baf-kserve-66f55ccfc9-cprfk\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") " pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:03.974793 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.974755    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-home\") pod \"test-model-dd4baf-kserve-66f55ccfc9-cprfk\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") " pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:03.974979 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.974799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-dshm\") pod \"test-model-dd4baf-kserve-66f55ccfc9-cprfk\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") " pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:03.974979 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.974825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68547b35-835e-483c-9f0b-0169e1ea1d74-tls-certs\") pod \"test-model-dd4baf-kserve-66f55ccfc9-cprfk\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") " pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:03.974979 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.974852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kfzx\" (UniqueName: \"kubernetes.io/projected/68547b35-835e-483c-9f0b-0169e1ea1d74-kube-api-access-5kfzx\") pod \"test-model-dd4baf-kserve-66f55ccfc9-cprfk\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") " pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:03.974979 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.974931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-tmp-dir\") pod \"test-model-dd4baf-kserve-66f55ccfc9-cprfk\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") " pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:03.975198 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.974996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-model-cache\") pod \"test-model-dd4baf-kserve-66f55ccfc9-cprfk\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") " pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:03.975198 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.975136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-home\") pod \"test-model-dd4baf-kserve-66f55ccfc9-cprfk\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") " pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:03.975337 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.975319    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-model-cache\") pod \"test-model-dd4baf-kserve-66f55ccfc9-cprfk\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") " pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:03.975394 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.975324    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-tmp-dir\") pod \"test-model-dd4baf-kserve-66f55ccfc9-cprfk\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") " pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:03.977223 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.977201    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-dshm\") pod \"test-model-dd4baf-kserve-66f55ccfc9-cprfk\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") " pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:03.977534 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.977519    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68547b35-835e-483c-9f0b-0169e1ea1d74-tls-certs\") pod \"test-model-dd4baf-kserve-66f55ccfc9-cprfk\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") " pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:03.983206 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:03.983184    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kfzx\" (UniqueName: \"kubernetes.io/projected/68547b35-835e-483c-9f0b-0169e1ea1d74-kube-api-access-5kfzx\") pod \"test-model-dd4baf-kserve-66f55ccfc9-cprfk\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") " pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:04.134306 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:04.134218    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:04.267811 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:04.267788    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"]
Jul 24 12:54:04.269874 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:54:04.269845    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68547b35_835e_483c_9f0b_0169e1ea1d74.slice/crio-604ba926d6b3648a8b2a1ae27ce2f3f2ebf6ef67bfc3ddf0c3c6429755589b6c WatchSource:0}: Error finding container 604ba926d6b3648a8b2a1ae27ce2f3f2ebf6ef67bfc3ddf0c3c6429755589b6c: Status 404 returned error can't find the container with id 604ba926d6b3648a8b2a1ae27ce2f3f2ebf6ef67bfc3ddf0c3c6429755589b6c
Jul 24 12:54:04.628574 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:04.628538    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk" event={"ID":"68547b35-835e-483c-9f0b-0169e1ea1d74","Type":"ContainerStarted","Data":"a33345aedc49c3b89cf7779384af7fad6aa75c23862ca37bb7594852bf3bf38f"}
Jul 24 12:54:04.628791 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:04.628586    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk" event={"ID":"68547b35-835e-483c-9f0b-0169e1ea1d74","Type":"ContainerStarted","Data":"604ba926d6b3648a8b2a1ae27ce2f3f2ebf6ef67bfc3ddf0c3c6429755589b6c"}
Jul 24 12:54:04.644963 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:04.644902    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk" podStartSLOduration=1.644883164 podStartE2EDuration="1.644883164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:04.644727762 +0000 UTC m=+2076.599732526" watchObservedRunningTime="2026-07-24 12:54:04.644883164 +0000 UTC m=+2076.599887927"
Jul 24 12:54:10.011809 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:10.011764    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:54:14.134755 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:14.134713    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:14.135183 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:14.134781    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:14.147222 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:14.147187    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:14.687597 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:14.687570    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:54:27.206932 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.206761    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"]
Jul 24 12:54:27.210385 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.210366    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:27.213162 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.213138    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fc0bc9\"/\"test-model-fc0bc9-kserve-self-signed-certs\""
Jul 24 12:54:27.213470 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.213450    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fc0bc9\"/\"kube-root-ca.crt\""
Jul 24 12:54:27.213562 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.213450    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fc0bc9\"/\"default-dockercfg-v6wqg\""
Jul 24 12:54:27.213796 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.213779    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fc0bc9\"/\"openshift-service-ca.crt\""
Jul 24 12:54:27.222591 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.222565    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"]
Jul 24 12:54:27.398809 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.398779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-tls-certs\") pod \"test-model-fc0bc9-kserve-55c797c8c5-cphvz\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") " pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:27.398809 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.398812    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-home\") pod \"test-model-fc0bc9-kserve-55c797c8c5-cphvz\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") " pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:27.399082 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.398836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-dshm\") pod \"test-model-fc0bc9-kserve-55c797c8c5-cphvz\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") " pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:27.399082 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.398859    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h5lp\" (UniqueName: \"kubernetes.io/projected/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-kube-api-access-9h5lp\") pod \"test-model-fc0bc9-kserve-55c797c8c5-cphvz\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") " pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:27.399082 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.398946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-model-cache\") pod \"test-model-fc0bc9-kserve-55c797c8c5-cphvz\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") " pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:27.399082 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.399065    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-tmp-dir\") pod \"test-model-fc0bc9-kserve-55c797c8c5-cphvz\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") " pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:27.500490 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.500377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-tmp-dir\") pod \"test-model-fc0bc9-kserve-55c797c8c5-cphvz\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") " pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:27.500490 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.500472    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-tls-certs\") pod \"test-model-fc0bc9-kserve-55c797c8c5-cphvz\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") " pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:27.500490 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.500490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-home\") pod \"test-model-fc0bc9-kserve-55c797c8c5-cphvz\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") " pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:27.500778 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.500512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-dshm\") pod \"test-model-fc0bc9-kserve-55c797c8c5-cphvz\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") " pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:27.500778 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.500538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h5lp\" (UniqueName: \"kubernetes.io/projected/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-kube-api-access-9h5lp\") pod \"test-model-fc0bc9-kserve-55c797c8c5-cphvz\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") " pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:27.500778 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.500562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-model-cache\") pod \"test-model-fc0bc9-kserve-55c797c8c5-cphvz\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") " pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:27.500932 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.500864    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-tmp-dir\") pod \"test-model-fc0bc9-kserve-55c797c8c5-cphvz\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") " pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:27.500932 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.500870    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-home\") pod \"test-model-fc0bc9-kserve-55c797c8c5-cphvz\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") " pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:27.500932 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.500912    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-model-cache\") pod \"test-model-fc0bc9-kserve-55c797c8c5-cphvz\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") " pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:27.502957 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.502936    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-dshm\") pod \"test-model-fc0bc9-kserve-55c797c8c5-cphvz\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") " pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:27.503198 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.503183    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-tls-certs\") pod \"test-model-fc0bc9-kserve-55c797c8c5-cphvz\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") " pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:27.509302 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.509283    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h5lp\" (UniqueName: \"kubernetes.io/projected/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-kube-api-access-9h5lp\") pod \"test-model-fc0bc9-kserve-55c797c8c5-cphvz\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") " pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:27.521167 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.521141    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:27.656394 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.656368    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"]
Jul 24 12:54:27.658165 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:54:27.658139    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3aeb2d8_59e5_43cd_aefb_3c24f0d61f95.slice/crio-9543274a9401a2fcc5b87c45ba42cce67570b579d7c6a750921157374b8352b8 WatchSource:0}: Error finding container 9543274a9401a2fcc5b87c45ba42cce67570b579d7c6a750921157374b8352b8: Status 404 returned error can't find the container with id 9543274a9401a2fcc5b87c45ba42cce67570b579d7c6a750921157374b8352b8
Jul 24 12:54:27.748798 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.748764    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz" event={"ID":"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95","Type":"ContainerStarted","Data":"2f410d5533571f41d85400e4844f3af1d251ad2db9ba0fd5391daf8e445cb9eb"}
Jul 24 12:54:27.748798 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.748806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz" event={"ID":"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95","Type":"ContainerStarted","Data":"9543274a9401a2fcc5b87c45ba42cce67570b579d7c6a750921157374b8352b8"}
Jul 24 12:54:27.765576 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:27.765475    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz" podStartSLOduration=0.765456967 podStartE2EDuration="765.456967ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:54:27.764819676 +0000 UTC m=+2099.719824440" watchObservedRunningTime="2026-07-24 12:54:27.765456967 +0000 UTC m=+2099.720461729"
Jul 24 12:54:28.740027 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:28.739997    2581 scope.go:117] "RemoveContainer" containerID="7b999701e7bc51bd9b8181fb75a842866a98908d954a6d21b620b9a0677c32ce"
Jul 24 12:54:32.907270 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:32.907241    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:54:37.521547 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:37.521509    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:37.522211 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:37.521571    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:37.534545 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:37.534515    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:54:37.810380 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:54:37.810351    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:55:20.376113 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:20.376023    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"]
Jul 24 12:55:21.271831 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:21.271800    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"]
Jul 24 12:55:25.002274 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:25.002239    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:55:35.396330 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:35.396272    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk" podUID="68547b35-835e-483c-9f0b-0169e1ea1d74" containerName="main" containerID="cri-o://a33345aedc49c3b89cf7779384af7fad6aa75c23862ca37bb7594852bf3bf38f" gracePeriod=45
Jul 24 12:55:35.654719 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:35.654654    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:55:35.722835 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:35.722796    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-model-cache\") pod \"68547b35-835e-483c-9f0b-0169e1ea1d74\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") "
Jul 24 12:55:35.722835 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:35.722838    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-dshm\") pod \"68547b35-835e-483c-9f0b-0169e1ea1d74\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") "
Jul 24 12:55:35.723086 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:35.722870    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68547b35-835e-483c-9f0b-0169e1ea1d74-tls-certs\") pod \"68547b35-835e-483c-9f0b-0169e1ea1d74\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") "
Jul 24 12:55:35.723086 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:35.722940    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-home\") pod \"68547b35-835e-483c-9f0b-0169e1ea1d74\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") "
Jul 24 12:55:35.723086 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:35.722975    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kfzx\" (UniqueName: \"kubernetes.io/projected/68547b35-835e-483c-9f0b-0169e1ea1d74-kube-api-access-5kfzx\") pod \"68547b35-835e-483c-9f0b-0169e1ea1d74\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") "
Jul 24 12:55:35.723086 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:35.723037    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-tmp-dir\") pod \"68547b35-835e-483c-9f0b-0169e1ea1d74\" (UID: \"68547b35-835e-483c-9f0b-0169e1ea1d74\") "
Jul 24 12:55:35.723272 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:35.723095    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-model-cache" (OuterVolumeSpecName: "model-cache") pod "68547b35-835e-483c-9f0b-0169e1ea1d74" (UID: "68547b35-835e-483c-9f0b-0169e1ea1d74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:55:35.723272 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:35.723221    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-home" (OuterVolumeSpecName: "home") pod "68547b35-835e-483c-9f0b-0169e1ea1d74" (UID: "68547b35-835e-483c-9f0b-0169e1ea1d74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:55:35.723379 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:35.723318    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "68547b35-835e-483c-9f0b-0169e1ea1d74" (UID: "68547b35-835e-483c-9f0b-0169e1ea1d74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:55:35.723462 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:35.723388    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-model-cache\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:55:35.723462 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:35.723406    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-home\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:55:35.725374 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:35.725350    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68547b35-835e-483c-9f0b-0169e1ea1d74-kube-api-access-5kfzx" (OuterVolumeSpecName: "kube-api-access-5kfzx") pod "68547b35-835e-483c-9f0b-0169e1ea1d74" (UID: "68547b35-835e-483c-9f0b-0169e1ea1d74"). InnerVolumeSpecName "kube-api-access-5kfzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:55:35.725374 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:35.725357    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-dshm" (OuterVolumeSpecName: "dshm") pod "68547b35-835e-483c-9f0b-0169e1ea1d74" (UID: "68547b35-835e-483c-9f0b-0169e1ea1d74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:55:35.725534 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:35.725372    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68547b35-835e-483c-9f0b-0169e1ea1d74-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68547b35-835e-483c-9f0b-0169e1ea1d74" (UID: "68547b35-835e-483c-9f0b-0169e1ea1d74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:55:35.824081 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:35.824048    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kfzx\" (UniqueName: \"kubernetes.io/projected/68547b35-835e-483c-9f0b-0169e1ea1d74-kube-api-access-5kfzx\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:55:35.824081 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:35.824076    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-tmp-dir\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:55:35.824081 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:35.824086    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68547b35-835e-483c-9f0b-0169e1ea1d74-dshm\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:55:35.824308 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:35.824095    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68547b35-835e-483c-9f0b-0169e1ea1d74-tls-certs\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:55:36.090161 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.090125    2581 generic.go:356] "Generic (PLEG): container finished" podID="68547b35-835e-483c-9f0b-0169e1ea1d74" containerID="a33345aedc49c3b89cf7779384af7fad6aa75c23862ca37bb7594852bf3bf38f" exitCode=0
Jul 24 12:55:36.090347 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.090169    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"
Jul 24 12:55:36.090347 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.090200    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk" event={"ID":"68547b35-835e-483c-9f0b-0169e1ea1d74","Type":"ContainerDied","Data":"a33345aedc49c3b89cf7779384af7fad6aa75c23862ca37bb7594852bf3bf38f"}
Jul 24 12:55:36.090347 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.090239    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk" event={"ID":"68547b35-835e-483c-9f0b-0169e1ea1d74","Type":"ContainerDied","Data":"604ba926d6b3648a8b2a1ae27ce2f3f2ebf6ef67bfc3ddf0c3c6429755589b6c"}
Jul 24 12:55:36.090347 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.090260    2581 scope.go:117] "RemoveContainer" containerID="a33345aedc49c3b89cf7779384af7fad6aa75c23862ca37bb7594852bf3bf38f"
Jul 24 12:55:36.099914 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.099892    2581 scope.go:117] "RemoveContainer" containerID="a33345aedc49c3b89cf7779384af7fad6aa75c23862ca37bb7594852bf3bf38f"
Jul 24 12:55:36.100173 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:55:36.100154    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a33345aedc49c3b89cf7779384af7fad6aa75c23862ca37bb7594852bf3bf38f\": container with ID starting with a33345aedc49c3b89cf7779384af7fad6aa75c23862ca37bb7594852bf3bf38f not found: ID does not exist" containerID="a33345aedc49c3b89cf7779384af7fad6aa75c23862ca37bb7594852bf3bf38f"
Jul 24 12:55:36.100213 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.100183    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a33345aedc49c3b89cf7779384af7fad6aa75c23862ca37bb7594852bf3bf38f"} err="failed to get container status \"a33345aedc49c3b89cf7779384af7fad6aa75c23862ca37bb7594852bf3bf38f\": rpc error: code = NotFound desc = could not find container \"a33345aedc49c3b89cf7779384af7fad6aa75c23862ca37bb7594852bf3bf38f\": container with ID starting with a33345aedc49c3b89cf7779384af7fad6aa75c23862ca37bb7594852bf3bf38f not found: ID does not exist"
Jul 24 12:55:36.113464 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.113431    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"]
Jul 24 12:55:36.116147 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.116124    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-dd4baf/test-model-dd4baf-kserve-66f55ccfc9-cprfk"]
Jul 24 12:55:36.291489 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.291432    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz" podUID="c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95" containerName="main" containerID="cri-o://2f410d5533571f41d85400e4844f3af1d251ad2db9ba0fd5391daf8e445cb9eb" gracePeriod=45
Jul 24 12:55:36.552322 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.552298    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:55:36.631726 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.631691    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-model-cache\") pod \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") "
Jul 24 12:55:36.631912 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.631746    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-tls-certs\") pod \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") "
Jul 24 12:55:36.631912 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.631764    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h5lp\" (UniqueName: \"kubernetes.io/projected/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-kube-api-access-9h5lp\") pod \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") "
Jul 24 12:55:36.631912 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.631832    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-dshm\") pod \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") "
Jul 24 12:55:36.631912 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.631873    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-home\") pod \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") "
Jul 24 12:55:36.631912 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.631898    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-tmp-dir\") pod \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\" (UID: \"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95\") "
Jul 24 12:55:36.632176 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.631970    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95" (UID: "c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:55:36.632229 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.632165    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-home" (OuterVolumeSpecName: "home") pod "c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95" (UID: "c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:55:36.632229 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.632180    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-model-cache\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:55:36.632339 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.632233    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95" (UID: "c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:55:36.634102 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.634075    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-dshm" (OuterVolumeSpecName: "dshm") pod "c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95" (UID: "c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 12:55:36.634207 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.634144    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95" (UID: "c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 12:55:36.634207 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.634156    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-kube-api-access-9h5lp" (OuterVolumeSpecName: "kube-api-access-9h5lp") pod "c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95" (UID: "c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95"). InnerVolumeSpecName "kube-api-access-9h5lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:55:36.653427 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.653377    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68547b35-835e-483c-9f0b-0169e1ea1d74" path="/var/lib/kubelet/pods/68547b35-835e-483c-9f0b-0169e1ea1d74/volumes"
Jul 24 12:55:36.733057 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.733016    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-dshm\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:55:36.733057 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.733050    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-home\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:55:36.733278 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.733090    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-tmp-dir\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:55:36.733278 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.733103    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-tls-certs\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:55:36.733278 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:36.733115    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h5lp\" (UniqueName: \"kubernetes.io/projected/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95-kube-api-access-9h5lp\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:55:37.097133 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:37.097099    2581 generic.go:356] "Generic (PLEG): container finished" podID="c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95" containerID="2f410d5533571f41d85400e4844f3af1d251ad2db9ba0fd5391daf8e445cb9eb" exitCode=0
Jul 24 12:55:37.097344 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:37.097154    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"
Jul 24 12:55:37.097344 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:37.097182    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz" event={"ID":"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95","Type":"ContainerDied","Data":"2f410d5533571f41d85400e4844f3af1d251ad2db9ba0fd5391daf8e445cb9eb"}
Jul 24 12:55:37.097344 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:37.097226    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz" event={"ID":"c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95","Type":"ContainerDied","Data":"9543274a9401a2fcc5b87c45ba42cce67570b579d7c6a750921157374b8352b8"}
Jul 24 12:55:37.097344 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:37.097249    2581 scope.go:117] "RemoveContainer" containerID="2f410d5533571f41d85400e4844f3af1d251ad2db9ba0fd5391daf8e445cb9eb"
Jul 24 12:55:37.105926 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:37.105879    2581 scope.go:117] "RemoveContainer" containerID="2f410d5533571f41d85400e4844f3af1d251ad2db9ba0fd5391daf8e445cb9eb"
Jul 24 12:55:37.106247 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:55:37.106227    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f410d5533571f41d85400e4844f3af1d251ad2db9ba0fd5391daf8e445cb9eb\": container with ID starting with 2f410d5533571f41d85400e4844f3af1d251ad2db9ba0fd5391daf8e445cb9eb not found: ID does not exist" containerID="2f410d5533571f41d85400e4844f3af1d251ad2db9ba0fd5391daf8e445cb9eb"
Jul 24 12:55:37.106321 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:37.106255    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f410d5533571f41d85400e4844f3af1d251ad2db9ba0fd5391daf8e445cb9eb"} err="failed to get container status \"2f410d5533571f41d85400e4844f3af1d251ad2db9ba0fd5391daf8e445cb9eb\": rpc error: code = NotFound desc = could not find container \"2f410d5533571f41d85400e4844f3af1d251ad2db9ba0fd5391daf8e445cb9eb\": container with ID starting with 2f410d5533571f41d85400e4844f3af1d251ad2db9ba0fd5391daf8e445cb9eb not found: ID does not exist"
Jul 24 12:55:37.126080 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:37.126043    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"]
Jul 24 12:55:37.130446 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:37.130401    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-fc0bc9/test-model-fc0bc9-kserve-55c797c8c5-cphvz"]
Jul 24 12:55:37.527106 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:37.527009    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:55:38.645544 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:38.645510    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95" path="/var/lib/kubelet/pods/c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95/volumes"
Jul 24 12:55:43.837454 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:43.837408    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc0bc9-bjrrh"]
Jul 24 12:55:43.837975 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:43.837809    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95" containerName="main"
Jul 24 12:55:43.837975 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:43.837822    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95" containerName="main"
Jul 24 12:55:43.837975 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:43.837837    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68547b35-835e-483c-9f0b-0169e1ea1d74" containerName="main"
Jul 24 12:55:43.837975 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:43.837843    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="68547b35-835e-483c-9f0b-0169e1ea1d74" containerName="main"
Jul 24 12:55:43.837975 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:43.837896    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3aeb2d8-59e5-43cd-aefb-3c24f0d61f95" containerName="main"
Jul 24 12:55:43.837975 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:43.837907    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="68547b35-835e-483c-9f0b-0169e1ea1d74" containerName="main"
Jul 24 12:55:43.842240 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:43.842222    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc0bc9-bjrrh"
Jul 24 12:55:43.844665 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:43.844631    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zw8hg\""
Jul 24 12:55:43.844968 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:43.844952    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 12:55:43.859543 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:43.859518    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc0bc9-bjrrh"]
Jul 24 12:55:43.899865 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:43.899827    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48cdfb75-d15f-4ceb-b54d-b118ccdff65c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fc0bc9-bjrrh\" (UID: \"48cdfb75-d15f-4ceb-b54d-b118ccdff65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc0bc9-bjrrh"
Jul 24 12:55:44.000824 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:44.000789    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48cdfb75-d15f-4ceb-b54d-b118ccdff65c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fc0bc9-bjrrh\" (UID: \"48cdfb75-d15f-4ceb-b54d-b118ccdff65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc0bc9-bjrrh"
Jul 24 12:55:44.001436 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:44.001394    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48cdfb75-d15f-4ceb-b54d-b118ccdff65c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fc0bc9-bjrrh\" (UID: \"48cdfb75-d15f-4ceb-b54d-b118ccdff65c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc0bc9-bjrrh"
Jul 24 12:55:44.152166 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:44.152082    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc0bc9-bjrrh"
Jul 24 12:55:44.296997 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:44.296974    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc0bc9-bjrrh"]
Jul 24 12:55:44.298781 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:55:44.298754    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cdfb75_d15f_4ceb_b54d_b118ccdff65c.slice/crio-2505d9921ac948da7b155766b2a23c0ea8f16af47a0eccdd2134f07578375939 WatchSource:0}: Error finding container 2505d9921ac948da7b155766b2a23c0ea8f16af47a0eccdd2134f07578375939: Status 404 returned error can't find the container with id 2505d9921ac948da7b155766b2a23c0ea8f16af47a0eccdd2134f07578375939
Jul 24 12:55:45.138515 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:45.138471    2581 generic.go:356] "Generic (PLEG): container finished" podID="48cdfb75-d15f-4ceb-b54d-b118ccdff65c" containerID="0838665646fcb4e268b05e15422895397d9fc8509399f0f9ec6ff854b3f66fd0" exitCode=0
Jul 24 12:55:45.138891 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:45.138571    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc0bc9-bjrrh" event={"ID":"48cdfb75-d15f-4ceb-b54d-b118ccdff65c","Type":"ContainerDied","Data":"0838665646fcb4e268b05e15422895397d9fc8509399f0f9ec6ff854b3f66fd0"}
Jul 24 12:55:45.138891 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:45.138636    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc0bc9-bjrrh" event={"ID":"48cdfb75-d15f-4ceb-b54d-b118ccdff65c","Type":"ContainerStarted","Data":"2505d9921ac948da7b155766b2a23c0ea8f16af47a0eccdd2134f07578375939"}
Jul 24 12:55:46.272580 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:46.272555    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc0bc9-bjrrh"
Jul 24 12:55:46.420842 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:46.420736    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48cdfb75-d15f-4ceb-b54d-b118ccdff65c-maas-api-service-ca\") pod \"48cdfb75-d15f-4ceb-b54d-b118ccdff65c\" (UID: \"48cdfb75-d15f-4ceb-b54d-b118ccdff65c\") "
Jul 24 12:55:46.421121 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:46.421097    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48cdfb75-d15f-4ceb-b54d-b118ccdff65c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48cdfb75-d15f-4ceb-b54d-b118ccdff65c" (UID: "48cdfb75-d15f-4ceb-b54d-b118ccdff65c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:55:46.521505 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:46.521465    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48cdfb75-d15f-4ceb-b54d-b118ccdff65c-maas-api-service-ca\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:55:47.150833 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:47.150793    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc0bc9-bjrrh"
Jul 24 12:55:47.150833 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:47.150820    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc0bc9-bjrrh" event={"ID":"48cdfb75-d15f-4ceb-b54d-b118ccdff65c","Type":"ContainerDied","Data":"2505d9921ac948da7b155766b2a23c0ea8f16af47a0eccdd2134f07578375939"}
Jul 24 12:55:47.151057 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:47.150874    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2505d9921ac948da7b155766b2a23c0ea8f16af47a0eccdd2134f07578375939"
Jul 24 12:55:57.795909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:57.795874    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:55:58.918531 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:58.918410    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc0bc9-bjrrh"]
Jul 24 12:55:58.934576 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:55:58.921101    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc0bc9-bjrrh"]
Jul 24 12:56:00.647579 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:00.647542    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48cdfb75-d15f-4ceb-b54d-b118ccdff65c" path="/var/lib/kubelet/pods/48cdfb75-d15f-4ceb-b54d-b118ccdff65c/volumes"
Jul 24 12:56:15.649017 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:15.648977    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:56:20.849155 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:20.849119    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2"]
Jul 24 12:56:20.849684 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:20.849671    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48cdfb75-d15f-4ceb-b54d-b118ccdff65c" containerName="revoke-keys"
Jul 24 12:56:20.849752 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:20.849691    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="48cdfb75-d15f-4ceb-b54d-b118ccdff65c" containerName="revoke-keys"
Jul 24 12:56:20.849843 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:20.849829    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="48cdfb75-d15f-4ceb-b54d-b118ccdff65c" containerName="revoke-keys"
Jul 24 12:56:20.852980 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:20.852959    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2"
Jul 24 12:56:20.861400 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:20.861380    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-aa0820\""
Jul 24 12:56:20.861526 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:20.861380    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-aa0820-dockercfg-782zv\""
Jul 24 12:56:20.892979 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:20.892948    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2"]
Jul 24 12:56:20.939231 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:20.939195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44cabf98-5f36-4f01-aa46-652a9b3ca4f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2\" (UID: \"44cabf98-5f36-4f01-aa46-652a9b3ca4f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2"
Jul 24 12:56:20.939471 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:20.939266    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdn58\" (UniqueName: \"kubernetes.io/projected/44cabf98-5f36-4f01-aa46-652a9b3ca4f9-kube-api-access-hdn58\") pod \"payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2\" (UID: \"44cabf98-5f36-4f01-aa46-652a9b3ca4f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2"
Jul 24 12:56:21.040500 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:21.040471    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44cabf98-5f36-4f01-aa46-652a9b3ca4f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2\" (UID: \"44cabf98-5f36-4f01-aa46-652a9b3ca4f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2"
Jul 24 12:56:21.040696 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:21.040545    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdn58\" (UniqueName: \"kubernetes.io/projected/44cabf98-5f36-4f01-aa46-652a9b3ca4f9-kube-api-access-hdn58\") pod \"payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2\" (UID: \"44cabf98-5f36-4f01-aa46-652a9b3ca4f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2"
Jul 24 12:56:21.041259 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:21.041234    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44cabf98-5f36-4f01-aa46-652a9b3ca4f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2\" (UID: \"44cabf98-5f36-4f01-aa46-652a9b3ca4f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2"
Jul 24 12:56:21.051437 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:21.051392    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdn58\" (UniqueName: \"kubernetes.io/projected/44cabf98-5f36-4f01-aa46-652a9b3ca4f9-kube-api-access-hdn58\") pod \"payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2\" (UID: \"44cabf98-5f36-4f01-aa46-652a9b3ca4f9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2"
Jul 24 12:56:21.162556 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:21.162470    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2"
Jul 24 12:56:21.346354 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:21.346323    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2"]
Jul 24 12:56:21.349923 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:56:21.349883    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44cabf98_5f36_4f01_aa46_652a9b3ca4f9.slice/crio-1482cf2223509797e111920bac48526c29d5fdca7ddff0c0a7dd55afd400e7f4 WatchSource:0}: Error finding container 1482cf2223509797e111920bac48526c29d5fdca7ddff0c0a7dd55afd400e7f4: Status 404 returned error can't find the container with id 1482cf2223509797e111920bac48526c29d5fdca7ddff0c0a7dd55afd400e7f4
Jul 24 12:56:21.351741 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:21.351725    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 12:56:22.323138 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:22.323094    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2" event={"ID":"44cabf98-5f36-4f01-aa46-652a9b3ca4f9","Type":"ContainerStarted","Data":"1482cf2223509797e111920bac48526c29d5fdca7ddff0c0a7dd55afd400e7f4"}
Jul 24 12:56:23.330895 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:23.330859    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2" event={"ID":"44cabf98-5f36-4f01-aa46-652a9b3ca4f9","Type":"ContainerStarted","Data":"e46c16afca53d2c815e456b4947a6a785cfb34a2ca833b706986b9976e807951"}
Jul 24 12:56:23.331342 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:23.330968    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2"
Jul 24 12:56:23.357781 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:23.357730    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2" podStartSLOduration=1.766682264 podStartE2EDuration="3.357714845s" totalImagesPullingTime="1.591032581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:56:23.35519859 +0000 UTC m=+2215.310203352" watchObservedRunningTime="2026-07-24 12:56:23.357714845 +0000 UTC m=+2215.312719586"
Jul 24 12:56:28.635452 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:28.635397    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:56:28.766743 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:28.766712    2581 scope.go:117] "RemoveContainer" containerID="0191d8d788c1b26a0a5600b06d50ea9132b044bfc1918efef33d76683557788e"
Jul 24 12:56:31.955938 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:31.955902    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f394b5-gjbjb"]
Jul 24 12:56:31.958985 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:31.958965    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f394b5-gjbjb"
Jul 24 12:56:31.961100 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:31.961073    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 12:56:31.961207 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:31.961074    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zw8hg\""
Jul 24 12:56:31.968388 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:31.968366    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f394b5-gjbjb"]
Jul 24 12:56:32.045115 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:32.045072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8fb847-8e91-4fdb-abf6-c684ba389aa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f394b5-gjbjb\" (UID: \"2f8fb847-8e91-4fdb-abf6-c684ba389aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f394b5-gjbjb"
Jul 24 12:56:32.146534 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:32.146502    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8fb847-8e91-4fdb-abf6-c684ba389aa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f394b5-gjbjb\" (UID: \"2f8fb847-8e91-4fdb-abf6-c684ba389aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f394b5-gjbjb"
Jul 24 12:56:32.147147 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:32.147120    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8fb847-8e91-4fdb-abf6-c684ba389aa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f394b5-gjbjb\" (UID: \"2f8fb847-8e91-4fdb-abf6-c684ba389aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f394b5-gjbjb"
Jul 24 12:56:32.269981 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:32.269904    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f394b5-gjbjb"
Jul 24 12:56:32.402621 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:32.402556    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f394b5-gjbjb"]
Jul 24 12:56:32.404967 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:56:32.404931    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8fb847_8e91_4fdb_abf6_c684ba389aa8.slice/crio-031ff5a166f54d36df9266cad54d97a25460faf4fde93f6b990028123b2bf974 WatchSource:0}: Error finding container 031ff5a166f54d36df9266cad54d97a25460faf4fde93f6b990028123b2bf974: Status 404 returned error can't find the container with id 031ff5a166f54d36df9266cad54d97a25460faf4fde93f6b990028123b2bf974
Jul 24 12:56:33.384678 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:33.384636    2581 generic.go:356] "Generic (PLEG): container finished" podID="2f8fb847-8e91-4fdb-abf6-c684ba389aa8" containerID="8ba9010534e6ba2096a1729dd962033d731920a06a51a5e19f882d028bc3f5af" exitCode=0
Jul 24 12:56:33.385070 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:33.384727    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f394b5-gjbjb" event={"ID":"2f8fb847-8e91-4fdb-abf6-c684ba389aa8","Type":"ContainerDied","Data":"8ba9010534e6ba2096a1729dd962033d731920a06a51a5e19f882d028bc3f5af"}
Jul 24 12:56:33.385070 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:33.384814    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f394b5-gjbjb" event={"ID":"2f8fb847-8e91-4fdb-abf6-c684ba389aa8","Type":"ContainerStarted","Data":"031ff5a166f54d36df9266cad54d97a25460faf4fde93f6b990028123b2bf974"}
Jul 24 12:56:34.338630 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:34.338593    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2"
Jul 24 12:56:34.527797 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:34.527773    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f394b5-gjbjb"
Jul 24 12:56:34.670724 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:34.670640    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8fb847-8e91-4fdb-abf6-c684ba389aa8-maas-api-service-ca\") pod \"2f8fb847-8e91-4fdb-abf6-c684ba389aa8\" (UID: \"2f8fb847-8e91-4fdb-abf6-c684ba389aa8\") "
Jul 24 12:56:34.670981 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:34.670959    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8fb847-8e91-4fdb-abf6-c684ba389aa8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2f8fb847-8e91-4fdb-abf6-c684ba389aa8" (UID: "2f8fb847-8e91-4fdb-abf6-c684ba389aa8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:56:34.771283 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:34.771249    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8fb847-8e91-4fdb-abf6-c684ba389aa8-maas-api-service-ca\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:56:35.395552 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:35.395522    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f394b5-gjbjb"
Jul 24 12:56:35.395703 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:35.395569    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f394b5-gjbjb" event={"ID":"2f8fb847-8e91-4fdb-abf6-c684ba389aa8","Type":"ContainerDied","Data":"031ff5a166f54d36df9266cad54d97a25460faf4fde93f6b990028123b2bf974"}
Jul 24 12:56:35.395703 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:35.395613    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="031ff5a166f54d36df9266cad54d97a25460faf4fde93f6b990028123b2bf974"
Jul 24 12:56:49.800710 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:49.800615    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:56:52.030810 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:52.030773    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f394b5-gjbjb"]
Jul 24 12:56:52.034854 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:52.034830    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f394b5-gjbjb"]
Jul 24 12:56:52.646252 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:52.646220    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f8fb847-8e91-4fdb-abf6-c684ba389aa8" path="/var/lib/kubelet/pods/2f8fb847-8e91-4fdb-abf6-c684ba389aa8/volumes"
Jul 24 12:56:59.362305 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:59.362271    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aa0820-nbsqs"]
Jul 24 12:56:59.362747 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:59.362675    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f8fb847-8e91-4fdb-abf6-c684ba389aa8" containerName="revoke-keys"
Jul 24 12:56:59.362747 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:59.362687    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f8fb847-8e91-4fdb-abf6-c684ba389aa8" containerName="revoke-keys"
Jul 24 12:56:59.362828 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:59.362761    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f8fb847-8e91-4fdb-abf6-c684ba389aa8" containerName="revoke-keys"
Jul 24 12:56:59.368128 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:59.368109    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aa0820-nbsqs"
Jul 24 12:56:59.370117 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:59.370095    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zw8hg\""
Jul 24 12:56:59.370250 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:59.370095    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 12:56:59.373892 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:59.373857    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aa0820-nbsqs"]
Jul 24 12:56:59.404371 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:59.404340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adbea749-613e-4d0e-b036-86e1bf9dab03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aa0820-nbsqs\" (UID: \"adbea749-613e-4d0e-b036-86e1bf9dab03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aa0820-nbsqs"
Jul 24 12:56:59.505003 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:59.504959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adbea749-613e-4d0e-b036-86e1bf9dab03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aa0820-nbsqs\" (UID: \"adbea749-613e-4d0e-b036-86e1bf9dab03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aa0820-nbsqs"
Jul 24 12:56:59.505631 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:59.505614    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adbea749-613e-4d0e-b036-86e1bf9dab03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-aa0820-nbsqs\" (UID: \"adbea749-613e-4d0e-b036-86e1bf9dab03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aa0820-nbsqs"
Jul 24 12:56:59.679864 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:59.679767    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aa0820-nbsqs"
Jul 24 12:56:59.815489 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:56:59.815463    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aa0820-nbsqs"]
Jul 24 12:56:59.819114 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:56:59.819085    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadbea749_613e_4d0e_b036_86e1bf9dab03.slice/crio-21258a75278cfd1bcd381f32d66a5a1ffb83f355806441941253261e56dc6733 WatchSource:0}: Error finding container 21258a75278cfd1bcd381f32d66a5a1ffb83f355806441941253261e56dc6733: Status 404 returned error can't find the container with id 21258a75278cfd1bcd381f32d66a5a1ffb83f355806441941253261e56dc6733
Jul 24 12:57:00.524290 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:00.524201    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aa0820-nbsqs" event={"ID":"adbea749-613e-4d0e-b036-86e1bf9dab03","Type":"ContainerStarted","Data":"d35e9b3a8ed6c42e8d28bdf7675592e7d27ecabdfaebeee3686b51fc7bf839bf"}
Jul 24 12:57:00.524290 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:00.524258    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aa0820-nbsqs" event={"ID":"adbea749-613e-4d0e-b036-86e1bf9dab03","Type":"ContainerStarted","Data":"21258a75278cfd1bcd381f32d66a5a1ffb83f355806441941253261e56dc6733"}
Jul 24 12:57:00.539993 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:00.539943    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aa0820-nbsqs" podStartSLOduration=1.5399264769999998 podStartE2EDuration="1.539926477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:57:00.537934399 +0000 UTC m=+2252.492939162" watchObservedRunningTime="2026-07-24 12:57:00.539926477 +0000 UTC m=+2252.494931239"
Jul 24 12:57:01.530047 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:01.530018    2581 generic.go:356] "Generic (PLEG): container finished" podID="adbea749-613e-4d0e-b036-86e1bf9dab03" containerID="d35e9b3a8ed6c42e8d28bdf7675592e7d27ecabdfaebeee3686b51fc7bf839bf" exitCode=0
Jul 24 12:57:01.530489 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:01.530114    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aa0820-nbsqs" event={"ID":"adbea749-613e-4d0e-b036-86e1bf9dab03","Type":"ContainerDied","Data":"d35e9b3a8ed6c42e8d28bdf7675592e7d27ecabdfaebeee3686b51fc7bf839bf"}
Jul 24 12:57:02.672476 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:02.672449    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aa0820-nbsqs"
Jul 24 12:57:02.735203 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:02.735171    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adbea749-613e-4d0e-b036-86e1bf9dab03-maas-api-service-ca\") pod \"adbea749-613e-4d0e-b036-86e1bf9dab03\" (UID: \"adbea749-613e-4d0e-b036-86e1bf9dab03\") "
Jul 24 12:57:02.735593 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:02.735568    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adbea749-613e-4d0e-b036-86e1bf9dab03-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "adbea749-613e-4d0e-b036-86e1bf9dab03" (UID: "adbea749-613e-4d0e-b036-86e1bf9dab03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:57:02.836776 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:02.836663    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adbea749-613e-4d0e-b036-86e1bf9dab03-maas-api-service-ca\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:57:03.542663 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:03.542628    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aa0820-nbsqs" event={"ID":"adbea749-613e-4d0e-b036-86e1bf9dab03","Type":"ContainerDied","Data":"21258a75278cfd1bcd381f32d66a5a1ffb83f355806441941253261e56dc6733"}
Jul 24 12:57:03.542663 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:03.542658    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aa0820-nbsqs"
Jul 24 12:57:03.542897 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:03.542686    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21258a75278cfd1bcd381f32d66a5a1ffb83f355806441941253261e56dc6733"
Jul 24 12:57:15.043975 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:15.043938    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2"]
Jul 24 12:57:15.044455 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:15.044194    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2" podUID="44cabf98-5f36-4f01-aa46-652a9b3ca4f9" containerName="payload-pre-processing" containerID="cri-o://e46c16afca53d2c815e456b4947a6a785cfb34a2ca833b706986b9976e807951" gracePeriod=30
Jul 24 12:57:15.308120 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:15.308049    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2"
Jul 24 12:57:15.455368 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:15.455329    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44cabf98-5f36-4f01-aa46-652a9b3ca4f9-plugins-config-volume\") pod \"44cabf98-5f36-4f01-aa46-652a9b3ca4f9\" (UID: \"44cabf98-5f36-4f01-aa46-652a9b3ca4f9\") "
Jul 24 12:57:15.455607 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:15.455480    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdn58\" (UniqueName: \"kubernetes.io/projected/44cabf98-5f36-4f01-aa46-652a9b3ca4f9-kube-api-access-hdn58\") pod \"44cabf98-5f36-4f01-aa46-652a9b3ca4f9\" (UID: \"44cabf98-5f36-4f01-aa46-652a9b3ca4f9\") "
Jul 24 12:57:15.455732 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:15.455708    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44cabf98-5f36-4f01-aa46-652a9b3ca4f9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "44cabf98-5f36-4f01-aa46-652a9b3ca4f9" (UID: "44cabf98-5f36-4f01-aa46-652a9b3ca4f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 12:57:15.457774 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:15.457749    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44cabf98-5f36-4f01-aa46-652a9b3ca4f9-kube-api-access-hdn58" (OuterVolumeSpecName: "kube-api-access-hdn58") pod "44cabf98-5f36-4f01-aa46-652a9b3ca4f9" (UID: "44cabf98-5f36-4f01-aa46-652a9b3ca4f9"). InnerVolumeSpecName "kube-api-access-hdn58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 12:57:15.556532 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:15.556493    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdn58\" (UniqueName: \"kubernetes.io/projected/44cabf98-5f36-4f01-aa46-652a9b3ca4f9-kube-api-access-hdn58\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:57:15.556532 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:15.556523    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44cabf98-5f36-4f01-aa46-652a9b3ca4f9-plugins-config-volume\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 12:57:15.605969 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:15.605880    2581 generic.go:356] "Generic (PLEG): container finished" podID="44cabf98-5f36-4f01-aa46-652a9b3ca4f9" containerID="e46c16afca53d2c815e456b4947a6a785cfb34a2ca833b706986b9976e807951" exitCode=0
Jul 24 12:57:15.605969 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:15.605944    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2"
Jul 24 12:57:15.606187 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:15.605967    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2" event={"ID":"44cabf98-5f36-4f01-aa46-652a9b3ca4f9","Type":"ContainerDied","Data":"e46c16afca53d2c815e456b4947a6a785cfb34a2ca833b706986b9976e807951"}
Jul 24 12:57:15.606187 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:15.606045    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2" event={"ID":"44cabf98-5f36-4f01-aa46-652a9b3ca4f9","Type":"ContainerDied","Data":"1482cf2223509797e111920bac48526c29d5fdca7ddff0c0a7dd55afd400e7f4"}
Jul 24 12:57:15.606187 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:15.606067    2581 scope.go:117] "RemoveContainer" containerID="e46c16afca53d2c815e456b4947a6a785cfb34a2ca833b706986b9976e807951"
Jul 24 12:57:15.615222 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:15.615201    2581 scope.go:117] "RemoveContainer" containerID="e46c16afca53d2c815e456b4947a6a785cfb34a2ca833b706986b9976e807951"
Jul 24 12:57:15.615537 ip-10-0-128-194 kubenswrapper[2581]: E0724 12:57:15.615518    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e46c16afca53d2c815e456b4947a6a785cfb34a2ca833b706986b9976e807951\": container with ID starting with e46c16afca53d2c815e456b4947a6a785cfb34a2ca833b706986b9976e807951 not found: ID does not exist" containerID="e46c16afca53d2c815e456b4947a6a785cfb34a2ca833b706986b9976e807951"
Jul 24 12:57:15.615616 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:15.615546    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e46c16afca53d2c815e456b4947a6a785cfb34a2ca833b706986b9976e807951"} err="failed to get container status \"e46c16afca53d2c815e456b4947a6a785cfb34a2ca833b706986b9976e807951\": rpc error: code = NotFound desc = could not find container \"e46c16afca53d2c815e456b4947a6a785cfb34a2ca833b706986b9976e807951\": container with ID starting with e46c16afca53d2c815e456b4947a6a785cfb34a2ca833b706986b9976e807951 not found: ID does not exist"
Jul 24 12:57:15.626544 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:15.626513    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2"]
Jul 24 12:57:15.632738 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:15.632713    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-aa0820-85cf96f9f6-m47k2"]
Jul 24 12:57:16.646065 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:16.646036    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44cabf98-5f36-4f01-aa46-652a9b3ca4f9" path="/var/lib/kubelet/pods/44cabf98-5f36-4f01-aa46-652a9b3ca4f9/volumes"
Jul 24 12:57:19.450455 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:19.450406    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aa0820-nbsqs"]
Jul 24 12:57:19.452442 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:19.452402    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-aa0820-nbsqs"]
Jul 24 12:57:20.397909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:20.397873    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:57:20.645469 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:20.645441    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adbea749-613e-4d0e-b036-86e1bf9dab03" path="/var/lib/kubelet/pods/adbea749-613e-4d0e-b036-86e1bf9dab03/volumes"
Jul 24 12:57:28.793660 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:28.793632    2581 scope.go:117] "RemoveContainer" containerID="f215a4a68b730f31583f84326cb1407d9614e4dfece52759e9b5259236697472"
Jul 24 12:57:31.109218 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.109185    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc"]
Jul 24 12:57:31.109599 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.109572    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adbea749-613e-4d0e-b036-86e1bf9dab03" containerName="revoke-keys"
Jul 24 12:57:31.109599 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.109584    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="adbea749-613e-4d0e-b036-86e1bf9dab03" containerName="revoke-keys"
Jul 24 12:57:31.109677 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.109601    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44cabf98-5f36-4f01-aa46-652a9b3ca4f9" containerName="payload-pre-processing"
Jul 24 12:57:31.109677 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.109607    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="44cabf98-5f36-4f01-aa46-652a9b3ca4f9" containerName="payload-pre-processing"
Jul 24 12:57:31.109677 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.109670    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="adbea749-613e-4d0e-b036-86e1bf9dab03" containerName="revoke-keys"
Jul 24 12:57:31.109795 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.109679    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="44cabf98-5f36-4f01-aa46-652a9b3ca4f9" containerName="payload-pre-processing"
Jul 24 12:57:31.114065 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.114045    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc"
Jul 24 12:57:31.116083 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.116065    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-d15f64\""
Jul 24 12:57:31.116323 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.116309    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-d15f64-dockercfg-s5846\""
Jul 24 12:57:31.123003 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.122981    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc"]
Jul 24 12:57:31.124909 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.124890    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh"]
Jul 24 12:57:31.128362 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.128348    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh"
Jul 24 12:57:31.141867 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.141840    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh"]
Jul 24 12:57:31.203577 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.203546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61290433-cc74-4be0-ad06-6d2fe7d65cb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc\" (UID: \"61290433-cc74-4be0-ad06-6d2fe7d65cb6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc"
Jul 24 12:57:31.203577 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.203583    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsqz2\" (UniqueName: \"kubernetes.io/projected/61290433-cc74-4be0-ad06-6d2fe7d65cb6-kube-api-access-wsqz2\") pod \"payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc\" (UID: \"61290433-cc74-4be0-ad06-6d2fe7d65cb6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc"
Jul 24 12:57:31.203842 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.203647    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh\" (UID: \"aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh"
Jul 24 12:57:31.203842 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.203669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsz68\" (UniqueName: \"kubernetes.io/projected/aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2-kube-api-access-nsz68\") pod \"payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh\" (UID: \"aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh"
Jul 24 12:57:31.304817 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.304783    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61290433-cc74-4be0-ad06-6d2fe7d65cb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc\" (UID: \"61290433-cc74-4be0-ad06-6d2fe7d65cb6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc"
Jul 24 12:57:31.304817 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.304821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsqz2\" (UniqueName: \"kubernetes.io/projected/61290433-cc74-4be0-ad06-6d2fe7d65cb6-kube-api-access-wsqz2\") pod \"payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc\" (UID: \"61290433-cc74-4be0-ad06-6d2fe7d65cb6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc"
Jul 24 12:57:31.305057 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.305011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh\" (UID: \"aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh"
Jul 24 12:57:31.305121 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.305069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsz68\" (UniqueName: \"kubernetes.io/projected/aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2-kube-api-access-nsz68\") pod \"payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh\" (UID: \"aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh"
Jul 24 12:57:31.305515 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.305498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61290433-cc74-4be0-ad06-6d2fe7d65cb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc\" (UID: \"61290433-cc74-4be0-ad06-6d2fe7d65cb6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc"
Jul 24 12:57:31.305729 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.305713    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh\" (UID: \"aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh"
Jul 24 12:57:31.314385 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.314363    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsz68\" (UniqueName: \"kubernetes.io/projected/aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2-kube-api-access-nsz68\") pod \"payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh\" (UID: \"aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh"
Jul 24 12:57:31.314525 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.314473    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsqz2\" (UniqueName: \"kubernetes.io/projected/61290433-cc74-4be0-ad06-6d2fe7d65cb6-kube-api-access-wsqz2\") pod \"payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc\" (UID: \"61290433-cc74-4be0-ad06-6d2fe7d65cb6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc"
Jul 24 12:57:31.425519 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.425405    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc"
Jul 24 12:57:31.437551 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.437526    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh"
Jul 24 12:57:31.573989 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.573959    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc"]
Jul 24 12:57:31.577502 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:57:31.577477    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61290433_cc74_4be0_ad06_6d2fe7d65cb6.slice/crio-9dd7de6bcb12f1453facb5b4ac21cc0ab808e588767299983b16037e9cae9647 WatchSource:0}: Error finding container 9dd7de6bcb12f1453facb5b4ac21cc0ab808e588767299983b16037e9cae9647: Status 404 returned error can't find the container with id 9dd7de6bcb12f1453facb5b4ac21cc0ab808e588767299983b16037e9cae9647
Jul 24 12:57:31.603865 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.603844    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh"]
Jul 24 12:57:31.619622 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:57:31.619593    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4bdcdb_418b_4b8d_b4a5_0cbe53bbcfa2.slice/crio-9d8b40c7ad31c602e01edd7325544af998caf7475c2a472510a19bba6f4a0cd3 WatchSource:0}: Error finding container 9d8b40c7ad31c602e01edd7325544af998caf7475c2a472510a19bba6f4a0cd3: Status 404 returned error can't find the container with id 9d8b40c7ad31c602e01edd7325544af998caf7475c2a472510a19bba6f4a0cd3
Jul 24 12:57:31.691327 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.691238    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh" event={"ID":"aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2","Type":"ContainerStarted","Data":"8201d26c801334a33b8bf6ee7ffbb2a292037e1c1b6d018266b5d67cacac0e9a"}
Jul 24 12:57:31.691327 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.691292    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh" event={"ID":"aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2","Type":"ContainerStarted","Data":"9d8b40c7ad31c602e01edd7325544af998caf7475c2a472510a19bba6f4a0cd3"}
Jul 24 12:57:31.691567 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.691344    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh"
Jul 24 12:57:31.693112 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.693089    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc" event={"ID":"61290433-cc74-4be0-ad06-6d2fe7d65cb6","Type":"ContainerStarted","Data":"97fb404b6a6027977ea3b318fc4ae36310e4a8954e4cf8621411382176e3f96d"}
Jul 24 12:57:31.693253 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.693137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc" event={"ID":"61290433-cc74-4be0-ad06-6d2fe7d65cb6","Type":"ContainerStarted","Data":"9dd7de6bcb12f1453facb5b4ac21cc0ab808e588767299983b16037e9cae9647"}
Jul 24 12:57:31.693253 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.693178    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc"
Jul 24 12:57:31.708472 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.708388    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh" podStartSLOduration=0.708373012 podStartE2EDuration="708.373012ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:57:31.70727908 +0000 UTC m=+2283.662283843" watchObservedRunningTime="2026-07-24 12:57:31.708373012 +0000 UTC m=+2283.663377808"
Jul 24 12:57:31.724291 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:31.724247    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc" podStartSLOduration=0.724232662 podStartE2EDuration="724.232662ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:57:31.722430788 +0000 UTC m=+2283.677435543" watchObservedRunningTime="2026-07-24 12:57:31.724232662 +0000 UTC m=+2283.679237403"
Jul 24 12:57:38.399948 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:38.399918    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:57:39.143262 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.143222    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8"]
Jul 24 12:57:39.147250 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.147224    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8"
Jul 24 12:57:39.148073 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.148051    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt"]
Jul 24 12:57:39.151059 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.151035    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-c6697a\""
Jul 24 12:57:39.151571 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.151553    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt"
Jul 24 12:57:39.153460 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.153352    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-c6697a-dockercfg-7pm2f\""
Jul 24 12:57:39.157277 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.157254    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8"]
Jul 24 12:57:39.170115 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.170073    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bdf7e41-145a-4816-a198-3037a4552393-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8\" (UID: \"5bdf7e41-145a-4816-a198-3037a4552393\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8"
Jul 24 12:57:39.170282 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.170177    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1efabaab-7c52-45a8-88ab-aa701f749d89-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt\" (UID: \"1efabaab-7c52-45a8-88ab-aa701f749d89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt"
Jul 24 12:57:39.170282 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.170219    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbm6z\" (UniqueName: \"kubernetes.io/projected/5bdf7e41-145a-4816-a198-3037a4552393-kube-api-access-fbm6z\") pod \"payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8\" (UID: \"5bdf7e41-145a-4816-a198-3037a4552393\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8"
Jul 24 12:57:39.170282 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.170256    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j52mp\" (UniqueName: \"kubernetes.io/projected/1efabaab-7c52-45a8-88ab-aa701f749d89-kube-api-access-j52mp\") pod \"payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt\" (UID: \"1efabaab-7c52-45a8-88ab-aa701f749d89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt"
Jul 24 12:57:39.171877 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.171854    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt"]
Jul 24 12:57:39.271352 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.271317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1efabaab-7c52-45a8-88ab-aa701f749d89-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt\" (UID: \"1efabaab-7c52-45a8-88ab-aa701f749d89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt"
Jul 24 12:57:39.271352 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.271356    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbm6z\" (UniqueName: \"kubernetes.io/projected/5bdf7e41-145a-4816-a198-3037a4552393-kube-api-access-fbm6z\") pod \"payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8\" (UID: \"5bdf7e41-145a-4816-a198-3037a4552393\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8"
Jul 24 12:57:39.271667 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.271378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j52mp\" (UniqueName: \"kubernetes.io/projected/1efabaab-7c52-45a8-88ab-aa701f749d89-kube-api-access-j52mp\") pod \"payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt\" (UID: \"1efabaab-7c52-45a8-88ab-aa701f749d89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt"
Jul 24 12:57:39.271667 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.271466    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bdf7e41-145a-4816-a198-3037a4552393-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8\" (UID: \"5bdf7e41-145a-4816-a198-3037a4552393\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8"
Jul 24 12:57:39.272112 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.272081    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1efabaab-7c52-45a8-88ab-aa701f749d89-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt\" (UID: \"1efabaab-7c52-45a8-88ab-aa701f749d89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt"
Jul 24 12:57:39.272542 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.272521    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bdf7e41-145a-4816-a198-3037a4552393-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8\" (UID: \"5bdf7e41-145a-4816-a198-3037a4552393\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8"
Jul 24 12:57:39.279949 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.279926    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbm6z\" (UniqueName: \"kubernetes.io/projected/5bdf7e41-145a-4816-a198-3037a4552393-kube-api-access-fbm6z\") pod \"payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8\" (UID: \"5bdf7e41-145a-4816-a198-3037a4552393\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8"
Jul 24 12:57:39.280076 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.280054    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j52mp\" (UniqueName: \"kubernetes.io/projected/1efabaab-7c52-45a8-88ab-aa701f749d89-kube-api-access-j52mp\") pod \"payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt\" (UID: \"1efabaab-7c52-45a8-88ab-aa701f749d89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt"
Jul 24 12:57:39.464481 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.464381    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8"
Jul 24 12:57:39.479771 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.479745    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt"
Jul 24 12:57:39.641116 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.640728    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8"]
Jul 24 12:57:39.642908 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:57:39.642861    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bdf7e41_145a_4816_a198_3037a4552393.slice/crio-264deff23fefb69f301ebe46e33ca902db3603e89858d7cfb8d6f3a013fdca61 WatchSource:0}: Error finding container 264deff23fefb69f301ebe46e33ca902db3603e89858d7cfb8d6f3a013fdca61: Status 404 returned error can't find the container with id 264deff23fefb69f301ebe46e33ca902db3603e89858d7cfb8d6f3a013fdca61
Jul 24 12:57:39.669492 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.669462    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt"]
Jul 24 12:57:39.679761 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:57:39.679723    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1efabaab_7c52_45a8_88ab_aa701f749d89.slice/crio-37434ccce1e2466c4c7d71e6c8fd981eab3a4c7333aa2d5a26ba37ddbc50648d WatchSource:0}: Error finding container 37434ccce1e2466c4c7d71e6c8fd981eab3a4c7333aa2d5a26ba37ddbc50648d: Status 404 returned error can't find the container with id 37434ccce1e2466c4c7d71e6c8fd981eab3a4c7333aa2d5a26ba37ddbc50648d
Jul 24 12:57:39.736633 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.736602    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt" event={"ID":"1efabaab-7c52-45a8-88ab-aa701f749d89","Type":"ContainerStarted","Data":"9d81f3f795316c75b92a0e1348df9128b6af661a427790939f5d6d14beb43062"}
Jul 24 12:57:39.736775 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.736665    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt"
Jul 24 12:57:39.736775 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.736696    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt" event={"ID":"1efabaab-7c52-45a8-88ab-aa701f749d89","Type":"ContainerStarted","Data":"37434ccce1e2466c4c7d71e6c8fd981eab3a4c7333aa2d5a26ba37ddbc50648d"}
Jul 24 12:57:39.738132 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.738108    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8" event={"ID":"5bdf7e41-145a-4816-a198-3037a4552393","Type":"ContainerStarted","Data":"c1b914b343b8bc144239fb67bab6044985a901ebd5eaadb50ee75dfba1dcb10d"}
Jul 24 12:57:39.738223 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.738158    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8" event={"ID":"5bdf7e41-145a-4816-a198-3037a4552393","Type":"ContainerStarted","Data":"264deff23fefb69f301ebe46e33ca902db3603e89858d7cfb8d6f3a013fdca61"}
Jul 24 12:57:39.738223 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.738208    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8"
Jul 24 12:57:39.753551 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.753482    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt" podStartSLOduration=0.75345908 podStartE2EDuration="753.45908ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:57:39.750749108 +0000 UTC m=+2291.705753866" watchObservedRunningTime="2026-07-24 12:57:39.75345908 +0000 UTC m=+2291.708463845"
Jul 24 12:57:39.766077 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:39.766027    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8" podStartSLOduration=0.766011455 podStartE2EDuration="766.011455ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:57:39.765165495 +0000 UTC m=+2291.720170263" watchObservedRunningTime="2026-07-24 12:57:39.766011455 +0000 UTC m=+2291.721016228"
Jul 24 12:57:42.700398 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:42.700368    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh"
Jul 24 12:57:42.700798 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:42.700565    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc"
Jul 24 12:57:49.599941 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:49.599908    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:57:50.746621 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:50.746582    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt"
Jul 24 12:57:50.747126 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:57:50.746663    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8"
Jul 24 12:58:03.118878 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.118844    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"]
Jul 24 12:58:03.122731 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.122705    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:03.125733 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.125557    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d15f64\"/\"default-dockercfg-rwpmz\""
Jul 24 12:58:03.125733 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.125605    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d15f64\"/\"kube-root-ca.crt\""
Jul 24 12:58:03.125733 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.125633    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d15f64\"/\"auth-test-model-d15f64-kserve-self-signed-certs\""
Jul 24 12:58:03.125733 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.125605    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d15f64\"/\"openshift-service-ca.crt\""
Jul 24 12:58:03.131547 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.131512    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"]
Jul 24 12:58:03.206528 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.206492    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-home\") pod \"auth-test-model-d15f64-kserve-55c644488c-znjgs\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") " pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:03.206749 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.206624    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-tmp-dir\") pod \"auth-test-model-d15f64-kserve-55c644488c-znjgs\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") " pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:03.206749 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.206650    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-model-cache\") pod \"auth-test-model-d15f64-kserve-55c644488c-znjgs\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") " pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:03.206749 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.206707    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-dshm\") pod \"auth-test-model-d15f64-kserve-55c644488c-znjgs\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") " pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:03.206894 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.206749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-tls-certs\") pod \"auth-test-model-d15f64-kserve-55c644488c-znjgs\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") " pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:03.206894 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.206778    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6hnq\" (UniqueName: \"kubernetes.io/projected/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-kube-api-access-z6hnq\") pod \"auth-test-model-d15f64-kserve-55c644488c-znjgs\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") " pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:03.308301 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.308257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-home\") pod \"auth-test-model-d15f64-kserve-55c644488c-znjgs\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") " pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:03.308546 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.308358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-tmp-dir\") pod \"auth-test-model-d15f64-kserve-55c644488c-znjgs\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") " pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:03.308546 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.308388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-model-cache\") pod \"auth-test-model-d15f64-kserve-55c644488c-znjgs\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") " pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:03.308546 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.308448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-dshm\") pod \"auth-test-model-d15f64-kserve-55c644488c-znjgs\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") " pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:03.308546 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.308480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-tls-certs\") pod \"auth-test-model-d15f64-kserve-55c644488c-znjgs\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") " pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:03.308546 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.308507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6hnq\" (UniqueName: \"kubernetes.io/projected/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-kube-api-access-z6hnq\") pod \"auth-test-model-d15f64-kserve-55c644488c-znjgs\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") " pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:03.308825 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.308771    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-home\") pod \"auth-test-model-d15f64-kserve-55c644488c-znjgs\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") " pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:03.308825 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.308781    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-tmp-dir\") pod \"auth-test-model-d15f64-kserve-55c644488c-znjgs\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") " pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:03.308928 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.308825    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-model-cache\") pod \"auth-test-model-d15f64-kserve-55c644488c-znjgs\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") " pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:03.311056 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.311027    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-dshm\") pod \"auth-test-model-d15f64-kserve-55c644488c-znjgs\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") " pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:03.311206 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.311187    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-tls-certs\") pod \"auth-test-model-d15f64-kserve-55c644488c-znjgs\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") " pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:03.317002 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.316979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6hnq\" (UniqueName: \"kubernetes.io/projected/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-kube-api-access-z6hnq\") pod \"auth-test-model-d15f64-kserve-55c644488c-znjgs\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") " pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:03.434812 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.434705    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:03.570125 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.570096    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"]
Jul 24 12:58:03.572165 ip-10-0-128-194 kubenswrapper[2581]: W0724 12:58:03.572136    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a3f7962_d4b2_462a_a31e_5f6853aeaa35.slice/crio-8fef525439129a02b82288f31e1a76e29e85250c24e873ea0d06b774ee4cb26a WatchSource:0}: Error finding container 8fef525439129a02b82288f31e1a76e29e85250c24e873ea0d06b774ee4cb26a: Status 404 returned error can't find the container with id 8fef525439129a02b82288f31e1a76e29e85250c24e873ea0d06b774ee4cb26a
Jul 24 12:58:03.874201 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.874170    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs" event={"ID":"3a3f7962-d4b2-462a-a31e-5f6853aeaa35","Type":"ContainerStarted","Data":"6ec223aeebafabaf523ebb64e57774632d685a44bc88640890c38bea1daf4b74"}
Jul 24 12:58:03.874376 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.874214    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs" event={"ID":"3a3f7962-d4b2-462a-a31e-5f6853aeaa35","Type":"ContainerStarted","Data":"8fef525439129a02b82288f31e1a76e29e85250c24e873ea0d06b774ee4cb26a"}
Jul 24 12:58:03.892737 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:03.892681    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs" podStartSLOduration=0.892655839 podStartE2EDuration="892.655839ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 12:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 12:58:03.89158447 +0000 UTC m=+2315.846589233" watchObservedRunningTime="2026-07-24 12:58:03.892655839 +0000 UTC m=+2315.847660601"
Jul 24 12:58:06.601713 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:06.601675    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:58:13.435021 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:13.434982    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:13.435530 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:13.435289    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:13.450399 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:13.450372    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:13.939946 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:13.939908    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 12:58:30.324737 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:30.324699    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:58:45.203067 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:58:45.203034    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:59:02.098548 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:59:02.098503    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:59:17.506288 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:59:17.506249    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:59:37.798968 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:59:37.798930    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 12:59:57.907060 ip-10-0-128-194 kubenswrapper[2581]: I0724 12:59:57.906974    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:00:00.138307 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:00.138275    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d15f64-29748300-66vdt"]
Jul 24 13:00:00.141976 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:00.141954    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d15f64-29748300-66vdt"
Jul 24 13:00:00.144084 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:00.144064    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zw8hg\""
Jul 24 13:00:00.147545 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:00.147515    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c6697a-29748300-gc72n"]
Jul 24 13:00:00.151674 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:00.151654    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748300-5ccxz"]
Jul 24 13:00:00.151817 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:00.151796    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c6697a-29748300-gc72n"
Jul 24 13:00:00.154249 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:00.154231    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d15f64-29748300-66vdt"
Jul 24 13:00:00.155484 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:00.155466    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d15f64-29748300-66vdt"]
Jul 24 13:00:00.155589 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:00.155567    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748300-5ccxz"
Jul 24 13:00:00.161006 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:00.160985    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c6697a-29748300-gc72n"]
Jul 24 13:00:00.180896 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:00.180829    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748300-5ccxz"]
Jul 24 13:00:00.192717 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:00.192693    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c6697a-29748300-gc72n"
Jul 24 13:00:00.201000 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:00.200978    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748300-5ccxz"
Jul 24 13:00:00.323665 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:00.323631    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d15f64-29748300-66vdt"]
Jul 24 13:00:00.326131 ip-10-0-128-194 kubenswrapper[2581]: W0724 13:00:00.326083    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b505b3_3c99_446c_80cf_73e2bb3bdcfd.slice/crio-32be69fb12eb14efda193a56bd6b83824d22b19e9e0560a7e2062ea0c5bc9827 WatchSource:0}: Error finding container 32be69fb12eb14efda193a56bd6b83824d22b19e9e0560a7e2062ea0c5bc9827: Status 404 returned error can't find the container with id 32be69fb12eb14efda193a56bd6b83824d22b19e9e0560a7e2062ea0c5bc9827
Jul 24 13:00:00.469881 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:00.469800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d15f64-29748300-66vdt" event={"ID":"f0b505b3-3c99-446c-80cf-73e2bb3bdcfd","Type":"ContainerStarted","Data":"32be69fb12eb14efda193a56bd6b83824d22b19e9e0560a7e2062ea0c5bc9827"}
Jul 24 13:00:00.593095 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:00.593071    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748300-5ccxz"]
Jul 24 13:00:00.595372 ip-10-0-128-194 kubenswrapper[2581]: W0724 13:00:00.595343    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b57764e_81e2_4d59_ae3b_a7b3466b8f83.slice/crio-3e1a9d2455d10d37076791d99cbab2dc4cc30dc0a889ee46f1ef35a320338c68 WatchSource:0}: Error finding container 3e1a9d2455d10d37076791d99cbab2dc4cc30dc0a889ee46f1ef35a320338c68: Status 404 returned error can't find the container with id 3e1a9d2455d10d37076791d99cbab2dc4cc30dc0a889ee46f1ef35a320338c68
Jul 24 13:00:00.608150 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:00.608125    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c6697a-29748300-gc72n"]
Jul 24 13:00:00.610343 ip-10-0-128-194 kubenswrapper[2581]: W0724 13:00:00.610311    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996e0a58_4ca8_41d7_b82e_fda2c909c539.slice/crio-5eb6c43c92fa9481335624c75e6b9a9833e8afdf406eac1a878813db3b7ec9ad WatchSource:0}: Error finding container 5eb6c43c92fa9481335624c75e6b9a9833e8afdf406eac1a878813db3b7ec9ad: Status 404 returned error can't find the container with id 5eb6c43c92fa9481335624c75e6b9a9833e8afdf406eac1a878813db3b7ec9ad
Jul 24 13:00:01.476723 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:01.476666    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c6697a-29748300-gc72n" event={"ID":"996e0a58-4ca8-41d7-b82e-fda2c909c539","Type":"ContainerStarted","Data":"03acc44d416a2159015bc3effff14a3e0c8d3bda935ed27d898adc687cf7a8d0"}
Jul 24 13:00:01.477188 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:01.476739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c6697a-29748300-gc72n" event={"ID":"996e0a58-4ca8-41d7-b82e-fda2c909c539","Type":"ContainerStarted","Data":"5eb6c43c92fa9481335624c75e6b9a9833e8afdf406eac1a878813db3b7ec9ad"}
Jul 24 13:00:01.478504 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:01.478472    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748300-5ccxz" event={"ID":"4b57764e-81e2-4d59-ae3b-a7b3466b8f83","Type":"ContainerStarted","Data":"4197c0c178e87205301a7f6d87af7f6e2991cbbfe77bff1753558f80297d51f3"}
Jul 24 13:00:01.478642 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:01.478524    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748300-5ccxz" event={"ID":"4b57764e-81e2-4d59-ae3b-a7b3466b8f83","Type":"ContainerStarted","Data":"3e1a9d2455d10d37076791d99cbab2dc4cc30dc0a889ee46f1ef35a320338c68"}
Jul 24 13:00:01.479953 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:01.479926    2581 generic.go:356] "Generic (PLEG): container finished" podID="f0b505b3-3c99-446c-80cf-73e2bb3bdcfd" containerID="4c741f4bc6b8f74190054a4003460bbbe9c411ebf44d755d4093bea0cf8bc78b" exitCode=0
Jul 24 13:00:01.480065 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:01.480006    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d15f64-29748300-66vdt" event={"ID":"f0b505b3-3c99-446c-80cf-73e2bb3bdcfd","Type":"ContainerDied","Data":"4c741f4bc6b8f74190054a4003460bbbe9c411ebf44d755d4093bea0cf8bc78b"}
Jul 24 13:00:01.492902 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:01.492845    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c6697a-29748300-gc72n" podStartSLOduration=1.492826045 podStartE2EDuration="1.492826045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:00:01.491396831 +0000 UTC m=+2433.446401595" watchObservedRunningTime="2026-07-24 13:00:01.492826045 +0000 UTC m=+2433.447830826"
Jul 24 13:00:01.519359 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:01.519285    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748300-5ccxz" podStartSLOduration=1.51926889 podStartE2EDuration="1.51926889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:00:01.51893002 +0000 UTC m=+2433.473934779" watchObservedRunningTime="2026-07-24 13:00:01.51926889 +0000 UTC m=+2433.474273653"
Jul 24 13:00:02.485795 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:02.485761    2581 generic.go:356] "Generic (PLEG): container finished" podID="996e0a58-4ca8-41d7-b82e-fda2c909c539" containerID="03acc44d416a2159015bc3effff14a3e0c8d3bda935ed27d898adc687cf7a8d0" exitCode=0
Jul 24 13:00:02.486234 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:02.485855    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c6697a-29748300-gc72n" event={"ID":"996e0a58-4ca8-41d7-b82e-fda2c909c539","Type":"ContainerDied","Data":"03acc44d416a2159015bc3effff14a3e0c8d3bda935ed27d898adc687cf7a8d0"}
Jul 24 13:00:02.487296 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:02.487275    2581 generic.go:356] "Generic (PLEG): container finished" podID="4b57764e-81e2-4d59-ae3b-a7b3466b8f83" containerID="4197c0c178e87205301a7f6d87af7f6e2991cbbfe77bff1753558f80297d51f3" exitCode=0
Jul 24 13:00:02.487370 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:02.487358    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748300-5ccxz" event={"ID":"4b57764e-81e2-4d59-ae3b-a7b3466b8f83","Type":"ContainerDied","Data":"4197c0c178e87205301a7f6d87af7f6e2991cbbfe77bff1753558f80297d51f3"}
Jul 24 13:00:02.625706 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:02.625684    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d15f64-29748300-66vdt"
Jul 24 13:00:03.493733 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:03.493690    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d15f64-29748300-66vdt" event={"ID":"f0b505b3-3c99-446c-80cf-73e2bb3bdcfd","Type":"ContainerDied","Data":"32be69fb12eb14efda193a56bd6b83824d22b19e9e0560a7e2062ea0c5bc9827"}
Jul 24 13:00:03.493733 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:03.493722    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d15f64-29748300-66vdt"
Jul 24 13:00:03.494333 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:03.493749    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32be69fb12eb14efda193a56bd6b83824d22b19e9e0560a7e2062ea0c5bc9827"
Jul 24 13:00:03.643730 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:03.643704    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748300-5ccxz"
Jul 24 13:00:03.687913 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:03.687889    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c6697a-29748300-gc72n"
Jul 24 13:00:04.499845 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:04.499819    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c6697a-29748300-gc72n"
Jul 24 13:00:04.500324 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:04.499857    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c6697a-29748300-gc72n" event={"ID":"996e0a58-4ca8-41d7-b82e-fda2c909c539","Type":"ContainerDied","Data":"5eb6c43c92fa9481335624c75e6b9a9833e8afdf406eac1a878813db3b7ec9ad"}
Jul 24 13:00:04.500324 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:04.499916    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5eb6c43c92fa9481335624c75e6b9a9833e8afdf406eac1a878813db3b7ec9ad"
Jul 24 13:00:04.501117 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:04.501090    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748300-5ccxz"
Jul 24 13:00:04.501117 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:04.501104    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748300-5ccxz" event={"ID":"4b57764e-81e2-4d59-ae3b-a7b3466b8f83","Type":"ContainerDied","Data":"3e1a9d2455d10d37076791d99cbab2dc4cc30dc0a889ee46f1ef35a320338c68"}
Jul 24 13:00:04.501297 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:04.501136    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e1a9d2455d10d37076791d99cbab2dc4cc30dc0a889ee46f1ef35a320338c68"
Jul 24 13:00:18.100865 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:18.100829    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:00:33.207742 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:33.207699    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"]
Jul 24 13:00:37.409456 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:37.409407    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:00:48.228503 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.228444    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs" podUID="3a3f7962-d4b2-462a-a31e-5f6853aeaa35" containerName="main" containerID="cri-o://6ec223aeebafabaf523ebb64e57774632d685a44bc88640890c38bea1daf4b74" gracePeriod=45
Jul 24 13:00:48.485350 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.485289    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 13:00:48.558375 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.558337    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-home\") pod \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") "
Jul 24 13:00:48.558582 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.558439    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-tls-certs\") pod \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") "
Jul 24 13:00:48.558582 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.558502    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6hnq\" (UniqueName: \"kubernetes.io/projected/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-kube-api-access-z6hnq\") pod \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") "
Jul 24 13:00:48.558706 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.558574    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-model-cache\") pod \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") "
Jul 24 13:00:48.558706 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.558648    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-home" (OuterVolumeSpecName: "home") pod "3a3f7962-d4b2-462a-a31e-5f6853aeaa35" (UID: "3a3f7962-d4b2-462a-a31e-5f6853aeaa35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:00:48.558817 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.558731    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-tmp-dir\") pod \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") "
Jul 24 13:00:48.558817 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.558759    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-dshm\") pod \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\" (UID: \"3a3f7962-d4b2-462a-a31e-5f6853aeaa35\") "
Jul 24 13:00:48.558817 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.558795    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a3f7962-d4b2-462a-a31e-5f6853aeaa35" (UID: "3a3f7962-d4b2-462a-a31e-5f6853aeaa35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:00:48.558970 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.558953    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a3f7962-d4b2-462a-a31e-5f6853aeaa35" (UID: "3a3f7962-d4b2-462a-a31e-5f6853aeaa35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:00:48.559173 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.559147    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-tmp-dir\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:00:48.559173 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.559169    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-home\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:00:48.559338 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.559181    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-model-cache\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:00:48.560886 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.560864    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a3f7962-d4b2-462a-a31e-5f6853aeaa35" (UID: "3a3f7962-d4b2-462a-a31e-5f6853aeaa35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:00:48.560886 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.560872    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-dshm" (OuterVolumeSpecName: "dshm") pod "3a3f7962-d4b2-462a-a31e-5f6853aeaa35" (UID: "3a3f7962-d4b2-462a-a31e-5f6853aeaa35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:00:48.561022 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.560868    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-kube-api-access-z6hnq" (OuterVolumeSpecName: "kube-api-access-z6hnq") pod "3a3f7962-d4b2-462a-a31e-5f6853aeaa35" (UID: "3a3f7962-d4b2-462a-a31e-5f6853aeaa35"). InnerVolumeSpecName "kube-api-access-z6hnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:00:48.659702 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.659680    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-tls-certs\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:00:48.659702 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.659703    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6hnq\" (UniqueName: \"kubernetes.io/projected/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-kube-api-access-z6hnq\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:00:48.659846 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.659713    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a3f7962-d4b2-462a-a31e-5f6853aeaa35-dshm\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:00:48.736461 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.736360    2581 generic.go:356] "Generic (PLEG): container finished" podID="3a3f7962-d4b2-462a-a31e-5f6853aeaa35" containerID="6ec223aeebafabaf523ebb64e57774632d685a44bc88640890c38bea1daf4b74" exitCode=0
Jul 24 13:00:48.736461 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.736403    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"
Jul 24 13:00:48.736646 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.736453    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs" event={"ID":"3a3f7962-d4b2-462a-a31e-5f6853aeaa35","Type":"ContainerDied","Data":"6ec223aeebafabaf523ebb64e57774632d685a44bc88640890c38bea1daf4b74"}
Jul 24 13:00:48.736646 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.736503    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs" event={"ID":"3a3f7962-d4b2-462a-a31e-5f6853aeaa35","Type":"ContainerDied","Data":"8fef525439129a02b82288f31e1a76e29e85250c24e873ea0d06b774ee4cb26a"}
Jul 24 13:00:48.736646 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.736520    2581 scope.go:117] "RemoveContainer" containerID="6ec223aeebafabaf523ebb64e57774632d685a44bc88640890c38bea1daf4b74"
Jul 24 13:00:48.745798 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.745776    2581 scope.go:117] "RemoveContainer" containerID="6ec223aeebafabaf523ebb64e57774632d685a44bc88640890c38bea1daf4b74"
Jul 24 13:00:48.746081 ip-10-0-128-194 kubenswrapper[2581]: E0724 13:00:48.746061    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec223aeebafabaf523ebb64e57774632d685a44bc88640890c38bea1daf4b74\": container with ID starting with 6ec223aeebafabaf523ebb64e57774632d685a44bc88640890c38bea1daf4b74 not found: ID does not exist" containerID="6ec223aeebafabaf523ebb64e57774632d685a44bc88640890c38bea1daf4b74"
Jul 24 13:00:48.746158 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.746091    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec223aeebafabaf523ebb64e57774632d685a44bc88640890c38bea1daf4b74"} err="failed to get container status \"6ec223aeebafabaf523ebb64e57774632d685a44bc88640890c38bea1daf4b74\": rpc error: code = NotFound desc = could not find container \"6ec223aeebafabaf523ebb64e57774632d685a44bc88640890c38bea1daf4b74\": container with ID starting with 6ec223aeebafabaf523ebb64e57774632d685a44bc88640890c38bea1daf4b74 not found: ID does not exist"
Jul 24 13:00:48.754623 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.754596    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"]
Jul 24 13:00:48.761264 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:48.761236    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d15f64/auth-test-model-d15f64-kserve-55c644488c-znjgs"]
Jul 24 13:00:50.646122 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:50.646088    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a3f7962-d4b2-462a-a31e-5f6853aeaa35" path="/var/lib/kubelet/pods/3a3f7962-d4b2-462a-a31e-5f6853aeaa35/volumes"
Jul 24 13:00:50.911859 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:00:50.911768    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:01:02.022737 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.022703    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"]
Jul 24 13:01:02.025098 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.023113    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b57764e-81e2-4d59-ae3b-a7b3466b8f83" containerName="cleanup"
Jul 24 13:01:02.025098 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.023125    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b57764e-81e2-4d59-ae3b-a7b3466b8f83" containerName="cleanup"
Jul 24 13:01:02.025098 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.023140    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a3f7962-d4b2-462a-a31e-5f6853aeaa35" containerName="main"
Jul 24 13:01:02.025098 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.023146    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a3f7962-d4b2-462a-a31e-5f6853aeaa35" containerName="main"
Jul 24 13:01:02.025098 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.023155    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0b505b3-3c99-446c-80cf-73e2bb3bdcfd" containerName="cleanup"
Jul 24 13:01:02.025098 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.023161    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0b505b3-3c99-446c-80cf-73e2bb3bdcfd" containerName="cleanup"
Jul 24 13:01:02.025098 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.023166    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="996e0a58-4ca8-41d7-b82e-fda2c909c539" containerName="cleanup"
Jul 24 13:01:02.025098 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.023171    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="996e0a58-4ca8-41d7-b82e-fda2c909c539" containerName="cleanup"
Jul 24 13:01:02.025098 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.023239    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a3f7962-d4b2-462a-a31e-5f6853aeaa35" containerName="main"
Jul 24 13:01:02.025098 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.023245    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0b505b3-3c99-446c-80cf-73e2bb3bdcfd" containerName="cleanup"
Jul 24 13:01:02.025098 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.023255    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b57764e-81e2-4d59-ae3b-a7b3466b8f83" containerName="cleanup"
Jul 24 13:01:02.025098 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.023268    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="996e0a58-4ca8-41d7-b82e-fda2c909c539" containerName="cleanup"
Jul 24 13:01:02.028659 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.028623    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:02.034263 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.034141    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d15f64\"/\"kube-root-ca.crt\""
Jul 24 13:01:02.034263 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.034207    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d15f64\"/\"sub-test-model-d15f64-kserve-self-signed-certs\""
Jul 24 13:01:02.034263 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.034211    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d15f64\"/\"default-dockercfg-rwpmz\""
Jul 24 13:01:02.034558 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.034215    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d15f64\"/\"openshift-service-ca.crt\""
Jul 24 13:01:02.039818 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.039794    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"]
Jul 24 13:01:02.090794 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.090758    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-tmp-dir\") pod \"sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") " pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:02.090794 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.090796    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-tls-certs\") pod \"sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") " pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:02.091032 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.090834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-home\") pod \"sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") " pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:02.091032 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.090870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-dshm\") pod \"sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") " pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:02.091032 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.090897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7gpd\" (UniqueName: \"kubernetes.io/projected/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-kube-api-access-x7gpd\") pod \"sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") " pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:02.091032 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.090937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-model-cache\") pod \"sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") " pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:02.191436 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.191359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-dshm\") pod \"sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") " pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:02.191436 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.191448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7gpd\" (UniqueName: \"kubernetes.io/projected/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-kube-api-access-x7gpd\") pod \"sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") " pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:02.191696 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.191497    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-model-cache\") pod \"sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") " pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:02.191696 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.191535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-tmp-dir\") pod \"sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") " pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:02.191696 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.191555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-tls-certs\") pod \"sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") " pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:02.191696 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.191600    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-home\") pod \"sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") " pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:02.191963 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.191931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-home\") pod \"sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") " pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:02.191963 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.191956    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-model-cache\") pod \"sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") " pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:02.192082 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.192004    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-tmp-dir\") pod \"sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") " pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:02.193931 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.193908    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-dshm\") pod \"sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") " pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:02.194080 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.194066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-tls-certs\") pod \"sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") " pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:02.199903 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.199884    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7gpd\" (UniqueName: \"kubernetes.io/projected/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-kube-api-access-x7gpd\") pod \"sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") " pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:02.343433 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.343379    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:02.489287 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.489263    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"]
Jul 24 13:01:02.491571 ip-10-0-128-194 kubenswrapper[2581]: W0724 13:01:02.491544    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcb19df2_5dfc_4ed7_a335_70cfcd3d1959.slice/crio-5572f95cc0fee856b2dc086c6e376c0dbc632cbdcec0423948bf52d8c9c5a8ab WatchSource:0}: Error finding container 5572f95cc0fee856b2dc086c6e376c0dbc632cbdcec0423948bf52d8c9c5a8ab: Status 404 returned error can't find the container with id 5572f95cc0fee856b2dc086c6e376c0dbc632cbdcec0423948bf52d8c9c5a8ab
Jul 24 13:01:02.813662 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.813621    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9" event={"ID":"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959","Type":"ContainerStarted","Data":"57f750e305c8b9428543631e108b5f1a27ef54474abadaf3c053bd6ef7e700fa"}
Jul 24 13:01:02.813881 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.813677    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9" event={"ID":"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959","Type":"ContainerStarted","Data":"5572f95cc0fee856b2dc086c6e376c0dbc632cbdcec0423948bf52d8c9c5a8ab"}
Jul 24 13:01:02.830925 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:02.830878    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9" podStartSLOduration=0.830861876 podStartE2EDuration="830.861876ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:01:02.830250013 +0000 UTC m=+2494.785254787" watchObservedRunningTime="2026-07-24 13:01:02.830861876 +0000 UTC m=+2494.785866639"
Jul 24 13:01:05.402059 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:05.402026    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:01:12.344052 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:12.344009    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:12.344560 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:12.344066    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:12.357732 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:12.357703    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:12.879525 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:12.879498    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:01:24.504063 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:24.504033    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:01:36.502069 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:36.502030    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:01:56.305807 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:56.305776    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"]
Jul 24 13:01:58.511116 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:01:58.511086    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:02:08.604857 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:08.604820    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:02:11.325502 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:11.325455    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9" podUID="bcb19df2-5dfc-4ed7-a335-70cfcd3d1959" containerName="main" containerID="cri-o://57f750e305c8b9428543631e108b5f1a27ef54474abadaf3c053bd6ef7e700fa" gracePeriod=45
Jul 24 13:02:11.583012 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:11.582946    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:02:11.765314 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:11.765273    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-dshm\") pod \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") "
Jul 24 13:02:11.765314 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:11.765323    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7gpd\" (UniqueName: \"kubernetes.io/projected/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-kube-api-access-x7gpd\") pod \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") "
Jul 24 13:02:11.765610 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:11.765347    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-model-cache\") pod \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") "
Jul 24 13:02:11.765610 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:11.765393    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-tls-certs\") pod \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") "
Jul 24 13:02:11.765610 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:11.765466    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-tmp-dir\") pod \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") "
Jul 24 13:02:11.765610 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:11.765500    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-home\") pod \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\" (UID: \"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959\") "
Jul 24 13:02:11.765835 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:11.765664    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-model-cache" (OuterVolumeSpecName: "model-cache") pod "bcb19df2-5dfc-4ed7-a335-70cfcd3d1959" (UID: "bcb19df2-5dfc-4ed7-a335-70cfcd3d1959"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:02:11.765835 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:11.765799    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bcb19df2-5dfc-4ed7-a335-70cfcd3d1959" (UID: "bcb19df2-5dfc-4ed7-a335-70cfcd3d1959"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:02:11.765835 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:11.765809    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-home" (OuterVolumeSpecName: "home") pod "bcb19df2-5dfc-4ed7-a335-70cfcd3d1959" (UID: "bcb19df2-5dfc-4ed7-a335-70cfcd3d1959"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:02:11.766244 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:11.766066    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-model-cache\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:02:11.766244 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:11.766094    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-tmp-dir\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:02:11.766244 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:11.766108    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-home\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:02:11.767667 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:11.767642    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-dshm" (OuterVolumeSpecName: "dshm") pod "bcb19df2-5dfc-4ed7-a335-70cfcd3d1959" (UID: "bcb19df2-5dfc-4ed7-a335-70cfcd3d1959"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:02:11.767761 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:11.767731    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-kube-api-access-x7gpd" (OuterVolumeSpecName: "kube-api-access-x7gpd") pod "bcb19df2-5dfc-4ed7-a335-70cfcd3d1959" (UID: "bcb19df2-5dfc-4ed7-a335-70cfcd3d1959"). InnerVolumeSpecName "kube-api-access-x7gpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:02:11.767761 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:11.767750    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bcb19df2-5dfc-4ed7-a335-70cfcd3d1959" (UID: "bcb19df2-5dfc-4ed7-a335-70cfcd3d1959"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:02:11.867583 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:11.867488    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-dshm\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:02:11.867583 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:11.867523    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7gpd\" (UniqueName: \"kubernetes.io/projected/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-kube-api-access-x7gpd\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:02:11.867583 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:11.867536    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959-tls-certs\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:02:12.180798 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:12.180720    2581 generic.go:356] "Generic (PLEG): container finished" podID="bcb19df2-5dfc-4ed7-a335-70cfcd3d1959" containerID="57f750e305c8b9428543631e108b5f1a27ef54474abadaf3c053bd6ef7e700fa" exitCode=0
Jul 24 13:02:12.180798 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:12.180760    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"
Jul 24 13:02:12.180982 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:12.180799    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9" event={"ID":"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959","Type":"ContainerDied","Data":"57f750e305c8b9428543631e108b5f1a27ef54474abadaf3c053bd6ef7e700fa"}
Jul 24 13:02:12.180982 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:12.180844    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9" event={"ID":"bcb19df2-5dfc-4ed7-a335-70cfcd3d1959","Type":"ContainerDied","Data":"5572f95cc0fee856b2dc086c6e376c0dbc632cbdcec0423948bf52d8c9c5a8ab"}
Jul 24 13:02:12.180982 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:12.180861    2581 scope.go:117] "RemoveContainer" containerID="57f750e305c8b9428543631e108b5f1a27ef54474abadaf3c053bd6ef7e700fa"
Jul 24 13:02:12.191364 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:12.191347    2581 scope.go:117] "RemoveContainer" containerID="57f750e305c8b9428543631e108b5f1a27ef54474abadaf3c053bd6ef7e700fa"
Jul 24 13:02:12.191722 ip-10-0-128-194 kubenswrapper[2581]: E0724 13:02:12.191694    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f750e305c8b9428543631e108b5f1a27ef54474abadaf3c053bd6ef7e700fa\": container with ID starting with 57f750e305c8b9428543631e108b5f1a27ef54474abadaf3c053bd6ef7e700fa not found: ID does not exist" containerID="57f750e305c8b9428543631e108b5f1a27ef54474abadaf3c053bd6ef7e700fa"
Jul 24 13:02:12.191770 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:12.191734    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f750e305c8b9428543631e108b5f1a27ef54474abadaf3c053bd6ef7e700fa"} err="failed to get container status \"57f750e305c8b9428543631e108b5f1a27ef54474abadaf3c053bd6ef7e700fa\": rpc error: code = NotFound desc = could not find container \"57f750e305c8b9428543631e108b5f1a27ef54474abadaf3c053bd6ef7e700fa\": container with ID starting with 57f750e305c8b9428543631e108b5f1a27ef54474abadaf3c053bd6ef7e700fa not found: ID does not exist"
Jul 24 13:02:12.202700 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:12.202672    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"]
Jul 24 13:02:12.206611 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:12.206587    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d15f64/sub-test-model-d15f64-kserve-7dbd8fbd59-c5ww9"]
Jul 24 13:02:12.645686 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:12.645651    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcb19df2-5dfc-4ed7-a335-70cfcd3d1959" path="/var/lib/kubelet/pods/bcb19df2-5dfc-4ed7-a335-70cfcd3d1959/volumes"
Jul 24 13:02:14.614493 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.614456    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"]
Jul 24 13:02:14.614888 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.614875    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcb19df2-5dfc-4ed7-a335-70cfcd3d1959" containerName="main"
Jul 24 13:02:14.614932 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.614890    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcb19df2-5dfc-4ed7-a335-70cfcd3d1959" containerName="main"
Jul 24 13:02:14.614973 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.614964    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcb19df2-5dfc-4ed7-a335-70cfcd3d1959" containerName="main"
Jul 24 13:02:14.619669 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.619645    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:14.624965 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.624942    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d15f64\"/\"openshift-service-ca.crt\""
Jul 24 13:02:14.625121 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.624982    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d15f64\"/\"rate-test-model-d15f64-kserve-self-signed-certs\""
Jul 24 13:02:14.625704 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.625686    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d15f64\"/\"default-dockercfg-rwpmz\""
Jul 24 13:02:14.625840 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.625688    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d15f64\"/\"kube-root-ca.crt\""
Jul 24 13:02:14.634654 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.634629    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"]
Jul 24 13:02:14.794328 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.794292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f9f4\" (UniqueName: \"kubernetes.io/projected/75bf4aed-b264-480c-92d7-1a04f845e76f-kube-api-access-2f9f4\") pod \"rate-test-model-d15f64-kserve-58f7c787bc-g4r64\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") " pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:14.794328 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.794328    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-model-cache\") pod \"rate-test-model-d15f64-kserve-58f7c787bc-g4r64\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") " pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:14.794584 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.794346    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75bf4aed-b264-480c-92d7-1a04f845e76f-tls-certs\") pod \"rate-test-model-d15f64-kserve-58f7c787bc-g4r64\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") " pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:14.794584 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.794369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-home\") pod \"rate-test-model-d15f64-kserve-58f7c787bc-g4r64\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") " pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:14.794584 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.794534    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-dshm\") pod \"rate-test-model-d15f64-kserve-58f7c787bc-g4r64\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") " pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:14.794584 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.794561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-tmp-dir\") pod \"rate-test-model-d15f64-kserve-58f7c787bc-g4r64\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") " pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:14.895167 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.895064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-dshm\") pod \"rate-test-model-d15f64-kserve-58f7c787bc-g4r64\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") " pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:14.895167 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.895117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-tmp-dir\") pod \"rate-test-model-d15f64-kserve-58f7c787bc-g4r64\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") " pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:14.895409 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.895212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9f4\" (UniqueName: \"kubernetes.io/projected/75bf4aed-b264-480c-92d7-1a04f845e76f-kube-api-access-2f9f4\") pod \"rate-test-model-d15f64-kserve-58f7c787bc-g4r64\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") " pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:14.895409 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.895235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-model-cache\") pod \"rate-test-model-d15f64-kserve-58f7c787bc-g4r64\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") " pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:14.895409 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.895256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75bf4aed-b264-480c-92d7-1a04f845e76f-tls-certs\") pod \"rate-test-model-d15f64-kserve-58f7c787bc-g4r64\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") " pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:14.895409 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.895288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-home\") pod \"rate-test-model-d15f64-kserve-58f7c787bc-g4r64\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") " pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:14.895622 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.895555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-tmp-dir\") pod \"rate-test-model-d15f64-kserve-58f7c787bc-g4r64\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") " pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:14.895659 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.895643    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-model-cache\") pod \"rate-test-model-d15f64-kserve-58f7c787bc-g4r64\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") " pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:14.895694 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.895669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-home\") pod \"rate-test-model-d15f64-kserve-58f7c787bc-g4r64\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") " pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:14.897477 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.897456    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-dshm\") pod \"rate-test-model-d15f64-kserve-58f7c787bc-g4r64\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") " pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:14.897848 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.897831    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75bf4aed-b264-480c-92d7-1a04f845e76f-tls-certs\") pod \"rate-test-model-d15f64-kserve-58f7c787bc-g4r64\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") " pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:14.903789 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.903769    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f9f4\" (UniqueName: \"kubernetes.io/projected/75bf4aed-b264-480c-92d7-1a04f845e76f-kube-api-access-2f9f4\") pod \"rate-test-model-d15f64-kserve-58f7c787bc-g4r64\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") " pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:14.930924 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:14.930885    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:15.068123 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:15.068099    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"]
Jul 24 13:02:15.073449 ip-10-0-128-194 kubenswrapper[2581]: W0724 13:02:15.070719    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75bf4aed_b264_480c_92d7_1a04f845e76f.slice/crio-b20b8a65113e004b4937a770f06a32796bf7ca2620219fe44b68eceb7d849006 WatchSource:0}: Error finding container b20b8a65113e004b4937a770f06a32796bf7ca2620219fe44b68eceb7d849006: Status 404 returned error can't find the container with id b20b8a65113e004b4937a770f06a32796bf7ca2620219fe44b68eceb7d849006
Jul 24 13:02:15.200183 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:15.200080    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64" event={"ID":"75bf4aed-b264-480c-92d7-1a04f845e76f","Type":"ContainerStarted","Data":"f10623a6630e93d6e9ee11b79ec0e082574e3ade4c6e7402698d7923c248e3d1"}
Jul 24 13:02:15.200183 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:15.200134    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64" event={"ID":"75bf4aed-b264-480c-92d7-1a04f845e76f","Type":"ContainerStarted","Data":"b20b8a65113e004b4937a770f06a32796bf7ca2620219fe44b68eceb7d849006"}
Jul 24 13:02:15.221615 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:15.221556    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64" podStartSLOduration=1.221537179 podStartE2EDuration="1.221537179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:02:15.218833984 +0000 UTC m=+2567.173838746" watchObservedRunningTime="2026-07-24 13:02:15.221537179 +0000 UTC m=+2567.176541942"
Jul 24 13:02:19.802659 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:19.802624    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:02:24.932041 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:24.932001    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:24.932511 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:24.932234    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:24.947518 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:24.947494    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:25.264811 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:25.264721    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:02:28.861137 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:28.861106    2581 scope.go:117] "RemoveContainer" containerID="0838665646fcb4e268b05e15422895397d9fc8509399f0f9ec6ff854b3f66fd0"
Jul 24 13:02:31.831266 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:31.831229    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"]
Jul 24 13:02:31.836678 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:31.836656    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:31.838926 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:31.838903    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6697a\"/\"openshift-service-ca.crt\""
Jul 24 13:02:31.839577 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:31.839555    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6697a\"/\"kube-root-ca.crt\""
Jul 24 13:02:31.839709 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:31.839579    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6697a\"/\"default-dockercfg-hk67r\""
Jul 24 13:02:31.839709 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:31.839584    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6697a\"/\"rate-test-model-c6697a-kserve-self-signed-certs\""
Jul 24 13:02:31.845651 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:31.845630    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"]
Jul 24 13:02:31.947807 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:31.947770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-tmp-dir\") pod \"rate-test-model-c6697a-kserve-85d6df994c-wb6mk\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") " pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:31.948016 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:31.947814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-dshm\") pod \"rate-test-model-c6697a-kserve-85d6df994c-wb6mk\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") " pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:31.948016 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:31.947903    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-home\") pod \"rate-test-model-c6697a-kserve-85d6df994c-wb6mk\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") " pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:31.948016 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:31.947955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zssc\" (UniqueName: \"kubernetes.io/projected/9357184f-082e-43ad-a082-2b57c3da62c5-kube-api-access-2zssc\") pod \"rate-test-model-c6697a-kserve-85d6df994c-wb6mk\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") " pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:31.948187 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:31.948082    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-model-cache\") pod \"rate-test-model-c6697a-kserve-85d6df994c-wb6mk\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") " pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:31.948187 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:31.948121    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9357184f-082e-43ad-a082-2b57c3da62c5-tls-certs\") pod \"rate-test-model-c6697a-kserve-85d6df994c-wb6mk\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") " pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:32.049169 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:32.049129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-tmp-dir\") pod \"rate-test-model-c6697a-kserve-85d6df994c-wb6mk\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") " pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:32.049169 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:32.049168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-dshm\") pod \"rate-test-model-c6697a-kserve-85d6df994c-wb6mk\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") " pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:32.049527 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:32.049205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-home\") pod \"rate-test-model-c6697a-kserve-85d6df994c-wb6mk\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") " pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:32.049527 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:32.049248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zssc\" (UniqueName: \"kubernetes.io/projected/9357184f-082e-43ad-a082-2b57c3da62c5-kube-api-access-2zssc\") pod \"rate-test-model-c6697a-kserve-85d6df994c-wb6mk\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") " pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:32.049527 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:32.049319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-model-cache\") pod \"rate-test-model-c6697a-kserve-85d6df994c-wb6mk\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") " pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:32.049527 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:32.049346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9357184f-082e-43ad-a082-2b57c3da62c5-tls-certs\") pod \"rate-test-model-c6697a-kserve-85d6df994c-wb6mk\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") " pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:32.049749 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:32.049558    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-tmp-dir\") pod \"rate-test-model-c6697a-kserve-85d6df994c-wb6mk\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") " pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:32.049749 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:32.049640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-home\") pod \"rate-test-model-c6697a-kserve-85d6df994c-wb6mk\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") " pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:32.049749 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:32.049744    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-model-cache\") pod \"rate-test-model-c6697a-kserve-85d6df994c-wb6mk\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") " pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:32.051576 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:32.051541    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-dshm\") pod \"rate-test-model-c6697a-kserve-85d6df994c-wb6mk\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") " pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:32.051865 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:32.051850    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9357184f-082e-43ad-a082-2b57c3da62c5-tls-certs\") pod \"rate-test-model-c6697a-kserve-85d6df994c-wb6mk\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") " pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:32.058714 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:32.058695    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zssc\" (UniqueName: \"kubernetes.io/projected/9357184f-082e-43ad-a082-2b57c3da62c5-kube-api-access-2zssc\") pod \"rate-test-model-c6697a-kserve-85d6df994c-wb6mk\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") " pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:32.147041 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:32.146948    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:32.288197 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:32.288114    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"]
Jul 24 13:02:32.290588 ip-10-0-128-194 kubenswrapper[2581]: W0724 13:02:32.290551    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9357184f_082e_43ad_a082_2b57c3da62c5.slice/crio-7986957be8e00c186f4c55e2753666900cbe1358a8faaa21cc84135404618e28 WatchSource:0}: Error finding container 7986957be8e00c186f4c55e2753666900cbe1358a8faaa21cc84135404618e28: Status 404 returned error can't find the container with id 7986957be8e00c186f4c55e2753666900cbe1358a8faaa21cc84135404618e28
Jul 24 13:02:33.296525 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:33.296489    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk" event={"ID":"9357184f-082e-43ad-a082-2b57c3da62c5","Type":"ContainerStarted","Data":"e126f1e21e902e11ddc1b2566ad2ed350330ec221e279626e1f167c59a77bd51"}
Jul 24 13:02:33.296949 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:33.296531    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk" event={"ID":"9357184f-082e-43ad-a082-2b57c3da62c5","Type":"ContainerStarted","Data":"7986957be8e00c186f4c55e2753666900cbe1358a8faaa21cc84135404618e28"}
Jul 24 13:02:33.317168 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:33.317111    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk" podStartSLOduration=2.317093436 podStartE2EDuration="2.317093436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:02:33.314752966 +0000 UTC m=+2585.269757729" watchObservedRunningTime="2026-07-24 13:02:33.317093436 +0000 UTC m=+2585.272098199"
Jul 24 13:02:35.107925 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:35.107890    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:02:42.147754 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:42.147714    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:42.148262 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:42.147810    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:42.160941 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:42.160917    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:02:42.353929 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:02:42.353888    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:03:04.401123 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:04.401033    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:03:27.705633 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:27.705593    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:03:28.886436 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:28.886387    2581 scope.go:117] "RemoveContainer" containerID="8ba9010534e6ba2096a1729dd962033d731920a06a51a5e19f882d028bc3f5af"
Jul 24 13:03:28.895001 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:28.894980    2581 scope.go:117] "RemoveContainer" containerID="d35e9b3a8ed6c42e8d28bdf7675592e7d27ecabdfaebeee3686b51fc7bf839bf"
Jul 24 13:03:31.509131 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:31.509096    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"]
Jul 24 13:03:32.300930 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:32.300895    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"]
Jul 24 13:03:43.209129 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:43.209091    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:03:46.529362 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.529303    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64" podUID="75bf4aed-b264-480c-92d7-1a04f845e76f" containerName="main" containerID="cri-o://f10623a6630e93d6e9ee11b79ec0e082574e3ade4c6e7402698d7923c248e3d1" gracePeriod=45
Jul 24 13:03:46.691219 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.691188    2581 generic.go:356] "Generic (PLEG): container finished" podID="75bf4aed-b264-480c-92d7-1a04f845e76f" containerID="f10623a6630e93d6e9ee11b79ec0e082574e3ade4c6e7402698d7923c248e3d1" exitCode=0
Jul 24 13:03:46.691398 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.691262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64" event={"ID":"75bf4aed-b264-480c-92d7-1a04f845e76f","Type":"ContainerDied","Data":"f10623a6630e93d6e9ee11b79ec0e082574e3ade4c6e7402698d7923c248e3d1"}
Jul 24 13:03:46.791734 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.791707    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:03:46.892388 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.892351    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-dshm\") pod \"75bf4aed-b264-480c-92d7-1a04f845e76f\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") "
Jul 24 13:03:46.892604 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.892401    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-home\") pod \"75bf4aed-b264-480c-92d7-1a04f845e76f\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") "
Jul 24 13:03:46.892604 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.892495    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-model-cache\") pod \"75bf4aed-b264-480c-92d7-1a04f845e76f\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") "
Jul 24 13:03:46.892604 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.892524    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75bf4aed-b264-480c-92d7-1a04f845e76f-tls-certs\") pod \"75bf4aed-b264-480c-92d7-1a04f845e76f\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") "
Jul 24 13:03:46.892604 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.892564    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-tmp-dir\") pod \"75bf4aed-b264-480c-92d7-1a04f845e76f\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") "
Jul 24 13:03:46.892814 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.892616    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f9f4\" (UniqueName: \"kubernetes.io/projected/75bf4aed-b264-480c-92d7-1a04f845e76f-kube-api-access-2f9f4\") pod \"75bf4aed-b264-480c-92d7-1a04f845e76f\" (UID: \"75bf4aed-b264-480c-92d7-1a04f845e76f\") "
Jul 24 13:03:46.892814 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.892727    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-home" (OuterVolumeSpecName: "home") pod "75bf4aed-b264-480c-92d7-1a04f845e76f" (UID: "75bf4aed-b264-480c-92d7-1a04f845e76f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:03:46.892931 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.892824    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-model-cache" (OuterVolumeSpecName: "model-cache") pod "75bf4aed-b264-480c-92d7-1a04f845e76f" (UID: "75bf4aed-b264-480c-92d7-1a04f845e76f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:03:46.892931 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.892857    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75bf4aed-b264-480c-92d7-1a04f845e76f" (UID: "75bf4aed-b264-480c-92d7-1a04f845e76f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:03:46.893039 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.893011    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-tmp-dir\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:03:46.893039 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.893031    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-home\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:03:46.893112 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.893045    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-model-cache\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:03:46.894756 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.894728    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-dshm" (OuterVolumeSpecName: "dshm") pod "75bf4aed-b264-480c-92d7-1a04f845e76f" (UID: "75bf4aed-b264-480c-92d7-1a04f845e76f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:03:46.894854 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.894810    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75bf4aed-b264-480c-92d7-1a04f845e76f-kube-api-access-2f9f4" (OuterVolumeSpecName: "kube-api-access-2f9f4") pod "75bf4aed-b264-480c-92d7-1a04f845e76f" (UID: "75bf4aed-b264-480c-92d7-1a04f845e76f"). InnerVolumeSpecName "kube-api-access-2f9f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:03:46.894854 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.894833    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75bf4aed-b264-480c-92d7-1a04f845e76f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75bf4aed-b264-480c-92d7-1a04f845e76f" (UID: "75bf4aed-b264-480c-92d7-1a04f845e76f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:03:46.993708 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.993668    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f9f4\" (UniqueName: \"kubernetes.io/projected/75bf4aed-b264-480c-92d7-1a04f845e76f-kube-api-access-2f9f4\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:03:46.993708 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.993701    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75bf4aed-b264-480c-92d7-1a04f845e76f-dshm\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:03:46.993708 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:46.993712    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75bf4aed-b264-480c-92d7-1a04f845e76f-tls-certs\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:03:47.320472 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.320432    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk" podUID="9357184f-082e-43ad-a082-2b57c3da62c5" containerName="main" containerID="cri-o://e126f1e21e902e11ddc1b2566ad2ed350330ec221e279626e1f167c59a77bd51" gracePeriod=45
Jul 24 13:03:47.585938 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.585861    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:03:47.698699 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.698668    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"
Jul 24 13:03:47.698911 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.698673    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64" event={"ID":"75bf4aed-b264-480c-92d7-1a04f845e76f","Type":"ContainerDied","Data":"b20b8a65113e004b4937a770f06a32796bf7ca2620219fe44b68eceb7d849006"}
Jul 24 13:03:47.698911 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.698813    2581 scope.go:117] "RemoveContainer" containerID="f10623a6630e93d6e9ee11b79ec0e082574e3ade4c6e7402698d7923c248e3d1"
Jul 24 13:03:47.699807 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.699778    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-home\") pod \"9357184f-082e-43ad-a082-2b57c3da62c5\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") "
Jul 24 13:03:47.699943 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.699832    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-dshm\") pod \"9357184f-082e-43ad-a082-2b57c3da62c5\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") "
Jul 24 13:03:47.699943 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.699908    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-tmp-dir\") pod \"9357184f-082e-43ad-a082-2b57c3da62c5\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") "
Jul 24 13:03:47.700063 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.699963    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-model-cache\") pod \"9357184f-082e-43ad-a082-2b57c3da62c5\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") "
Jul 24 13:03:47.700063 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.699992    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9357184f-082e-43ad-a082-2b57c3da62c5-tls-certs\") pod \"9357184f-082e-43ad-a082-2b57c3da62c5\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") "
Jul 24 13:03:47.700063 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.700026    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zssc\" (UniqueName: \"kubernetes.io/projected/9357184f-082e-43ad-a082-2b57c3da62c5-kube-api-access-2zssc\") pod \"9357184f-082e-43ad-a082-2b57c3da62c5\" (UID: \"9357184f-082e-43ad-a082-2b57c3da62c5\") "
Jul 24 13:03:47.700297 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.700108    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-home" (OuterVolumeSpecName: "home") pod "9357184f-082e-43ad-a082-2b57c3da62c5" (UID: "9357184f-082e-43ad-a082-2b57c3da62c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:03:47.700358 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.700296    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "9357184f-082e-43ad-a082-2b57c3da62c5" (UID: "9357184f-082e-43ad-a082-2b57c3da62c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:03:47.700358 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.700305    2581 generic.go:356] "Generic (PLEG): container finished" podID="9357184f-082e-43ad-a082-2b57c3da62c5" containerID="e126f1e21e902e11ddc1b2566ad2ed350330ec221e279626e1f167c59a77bd51" exitCode=0
Jul 24 13:03:47.700358 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.700341    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk" event={"ID":"9357184f-082e-43ad-a082-2b57c3da62c5","Type":"ContainerDied","Data":"e126f1e21e902e11ddc1b2566ad2ed350330ec221e279626e1f167c59a77bd51"}
Jul 24 13:03:47.700358 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.700348    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-home\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:03:47.700599 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.700373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk" event={"ID":"9357184f-082e-43ad-a082-2b57c3da62c5","Type":"ContainerDied","Data":"7986957be8e00c186f4c55e2753666900cbe1358a8faaa21cc84135404618e28"}
Jul 24 13:03:47.700599 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.700454    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"
Jul 24 13:03:47.700714 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.700646    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9357184f-082e-43ad-a082-2b57c3da62c5" (UID: "9357184f-082e-43ad-a082-2b57c3da62c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:03:47.702617 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.702593    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-dshm" (OuterVolumeSpecName: "dshm") pod "9357184f-082e-43ad-a082-2b57c3da62c5" (UID: "9357184f-082e-43ad-a082-2b57c3da62c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:03:47.702846 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.702825    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9357184f-082e-43ad-a082-2b57c3da62c5-kube-api-access-2zssc" (OuterVolumeSpecName: "kube-api-access-2zssc") pod "9357184f-082e-43ad-a082-2b57c3da62c5" (UID: "9357184f-082e-43ad-a082-2b57c3da62c5"). InnerVolumeSpecName "kube-api-access-2zssc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:03:47.702966 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.702950    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9357184f-082e-43ad-a082-2b57c3da62c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9357184f-082e-43ad-a082-2b57c3da62c5" (UID: "9357184f-082e-43ad-a082-2b57c3da62c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:03:47.721829 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.721805    2581 scope.go:117] "RemoveContainer" containerID="e126f1e21e902e11ddc1b2566ad2ed350330ec221e279626e1f167c59a77bd51"
Jul 24 13:03:47.730171 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.730142    2581 scope.go:117] "RemoveContainer" containerID="e126f1e21e902e11ddc1b2566ad2ed350330ec221e279626e1f167c59a77bd51"
Jul 24 13:03:47.730856 ip-10-0-128-194 kubenswrapper[2581]: E0724 13:03:47.730823    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e126f1e21e902e11ddc1b2566ad2ed350330ec221e279626e1f167c59a77bd51\": container with ID starting with e126f1e21e902e11ddc1b2566ad2ed350330ec221e279626e1f167c59a77bd51 not found: ID does not exist" containerID="e126f1e21e902e11ddc1b2566ad2ed350330ec221e279626e1f167c59a77bd51"
Jul 24 13:03:47.730942 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.730866    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e126f1e21e902e11ddc1b2566ad2ed350330ec221e279626e1f167c59a77bd51"} err="failed to get container status \"e126f1e21e902e11ddc1b2566ad2ed350330ec221e279626e1f167c59a77bd51\": rpc error: code = NotFound desc = could not find container \"e126f1e21e902e11ddc1b2566ad2ed350330ec221e279626e1f167c59a77bd51\": container with ID starting with e126f1e21e902e11ddc1b2566ad2ed350330ec221e279626e1f167c59a77bd51 not found: ID does not exist"
Jul 24 13:03:47.732757 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.732735    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"]
Jul 24 13:03:47.736871 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.736849    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d15f64/rate-test-model-d15f64-kserve-58f7c787bc-g4r64"]
Jul 24 13:03:47.801178 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.801133    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-tmp-dir\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:03:47.801178 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.801170    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-model-cache\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:03:47.801178 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.801182    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9357184f-082e-43ad-a082-2b57c3da62c5-tls-certs\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:03:47.801463 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.801196    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zssc\" (UniqueName: \"kubernetes.io/projected/9357184f-082e-43ad-a082-2b57c3da62c5-kube-api-access-2zssc\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:03:47.801463 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:47.801208    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9357184f-082e-43ad-a082-2b57c3da62c5-dshm\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:03:48.021336 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:48.021300    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"]
Jul 24 13:03:48.023903 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:48.023878    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c6697a/rate-test-model-c6697a-kserve-85d6df994c-wb6mk"]
Jul 24 13:03:48.558347 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:48.558310    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7904-140605933188560"]
Jul 24 13:03:48.558745 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:48.558731    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75bf4aed-b264-480c-92d7-1a04f845e76f" containerName="main"
Jul 24 13:03:48.558798 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:48.558746    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="75bf4aed-b264-480c-92d7-1a04f845e76f" containerName="main"
Jul 24 13:03:48.558798 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:48.558764    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9357184f-082e-43ad-a082-2b57c3da62c5" containerName="main"
Jul 24 13:03:48.558798 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:48.558769    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9357184f-082e-43ad-a082-2b57c3da62c5" containerName="main"
Jul 24 13:03:48.558897 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:48.558823    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9357184f-082e-43ad-a082-2b57c3da62c5" containerName="main"
Jul 24 13:03:48.558897 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:48.558833    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="75bf4aed-b264-480c-92d7-1a04f845e76f" containerName="main"
Jul 24 13:03:48.563128 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:48.563111    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7904-140605933188560"
Jul 24 13:03:48.565887 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:48.565830    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xg2cx\""
Jul 24 13:03:48.567925 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:48.567899    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7904-140605933188560"]
Jul 24 13:03:48.608370 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:48.608334    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdfxs\" (UniqueName: \"kubernetes.io/projected/c6748b5f-5054-442a-8989-989f157db936-kube-api-access-kdfxs\") pod \"test-curl-7904-140605933188560\" (UID: \"c6748b5f-5054-442a-8989-989f157db936\") " pod="opendatahub/test-curl-7904-140605933188560"
Jul 24 13:03:48.647792 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:48.647761    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75bf4aed-b264-480c-92d7-1a04f845e76f" path="/var/lib/kubelet/pods/75bf4aed-b264-480c-92d7-1a04f845e76f/volumes"
Jul 24 13:03:48.648151 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:48.648138    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9357184f-082e-43ad-a082-2b57c3da62c5" path="/var/lib/kubelet/pods/9357184f-082e-43ad-a082-2b57c3da62c5/volumes"
Jul 24 13:03:48.709088 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:48.709051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdfxs\" (UniqueName: \"kubernetes.io/projected/c6748b5f-5054-442a-8989-989f157db936-kube-api-access-kdfxs\") pod \"test-curl-7904-140605933188560\" (UID: \"c6748b5f-5054-442a-8989-989f157db936\") " pod="opendatahub/test-curl-7904-140605933188560"
Jul 24 13:03:48.717765 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:48.717736    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdfxs\" (UniqueName: \"kubernetes.io/projected/c6748b5f-5054-442a-8989-989f157db936-kube-api-access-kdfxs\") pod \"test-curl-7904-140605933188560\" (UID: \"c6748b5f-5054-442a-8989-989f157db936\") " pod="opendatahub/test-curl-7904-140605933188560"
Jul 24 13:03:48.873451 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:48.873337    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7904-140605933188560"
Jul 24 13:03:49.015059 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:49.015037    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7904-140605933188560"]
Jul 24 13:03:49.017235 ip-10-0-128-194 kubenswrapper[2581]: W0724 13:03:49.017207    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6748b5f_5054_442a_8989_989f157db936.slice/crio-e7adf6151e7d80a62eb19fb3073a033dcd405f7aaa283b9f527abb85ddac2dd9 WatchSource:0}: Error finding container e7adf6151e7d80a62eb19fb3073a033dcd405f7aaa283b9f527abb85ddac2dd9: Status 404 returned error can't find the container with id e7adf6151e7d80a62eb19fb3073a033dcd405f7aaa283b9f527abb85ddac2dd9
Jul 24 13:03:49.018550 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:49.018532    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:03:49.720546 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:49.720516    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7904-140605933188560" event={"ID":"c6748b5f-5054-442a-8989-989f157db936","Type":"ContainerStarted","Data":"e7adf6151e7d80a62eb19fb3073a033dcd405f7aaa283b9f527abb85ddac2dd9"}
Jul 24 13:03:50.727466 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:50.727407    2581 generic.go:356] "Generic (PLEG): container finished" podID="c6748b5f-5054-442a-8989-989f157db936" containerID="51b515643ef62da7de00bd8d9f09f806b0c119c8059043482089bf8e098d1639" exitCode=0
Jul 24 13:03:50.727861 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:50.727521    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7904-140605933188560" event={"ID":"c6748b5f-5054-442a-8989-989f157db936","Type":"ContainerDied","Data":"51b515643ef62da7de00bd8d9f09f806b0c119c8059043482089bf8e098d1639"}
Jul 24 13:03:51.873537 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:51.873508    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7904-140605933188560"
Jul 24 13:03:51.908722 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:51.908694    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7904-140605933188560_c6748b5f-5054-442a-8989-989f157db936/test-curl-7904-140605933188560/0.log"
Jul 24 13:03:51.933174 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:51.933138    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7904-140605933188560"]
Jul 24 13:03:51.937357 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:51.937328    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7904-140605933188560"]
Jul 24 13:03:51.940483 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:51.940463    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdfxs\" (UniqueName: \"kubernetes.io/projected/c6748b5f-5054-442a-8989-989f157db936-kube-api-access-kdfxs\") pod \"c6748b5f-5054-442a-8989-989f157db936\" (UID: \"c6748b5f-5054-442a-8989-989f157db936\") "
Jul 24 13:03:51.942830 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:51.942807    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6748b5f-5054-442a-8989-989f157db936-kube-api-access-kdfxs" (OuterVolumeSpecName: "kube-api-access-kdfxs") pod "c6748b5f-5054-442a-8989-989f157db936" (UID: "c6748b5f-5054-442a-8989-989f157db936"). InnerVolumeSpecName "kube-api-access-kdfxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:03:52.041601 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:52.041569    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdfxs\" (UniqueName: \"kubernetes.io/projected/c6748b5f-5054-442a-8989-989f157db936-kube-api-access-kdfxs\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:03:52.647079 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:52.647047    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6748b5f-5054-442a-8989-989f157db936" path="/var/lib/kubelet/pods/c6748b5f-5054-442a-8989-989f157db936/volumes"
Jul 24 13:03:52.739844 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:52.739811    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7904-140605933188560"
Jul 24 13:03:52.740008 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:52.739866    2581 scope.go:117] "RemoveContainer" containerID="51b515643ef62da7de00bd8d9f09f806b0c119c8059043482089bf8e098d1639"
Jul 24 13:03:57.233570 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:57.233531    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz"]
Jul 24 13:03:57.233961 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:57.233928    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6748b5f-5054-442a-8989-989f157db936" containerName="test-curl-7904-140605933188560"
Jul 24 13:03:57.233961 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:57.233940    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6748b5f-5054-442a-8989-989f157db936" containerName="test-curl-7904-140605933188560"
Jul 24 13:03:57.234033 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:57.234012    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6748b5f-5054-442a-8989-989f157db936" containerName="test-curl-7904-140605933188560"
Jul 24 13:03:57.238659 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:57.238635    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz"
Jul 24 13:03:57.240951 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:57.240923    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-967d3e\""
Jul 24 13:03:57.241077 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:57.240955    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-967d3e-dockercfg-wrgrq\""
Jul 24 13:03:57.248522 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:57.248496    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz"]
Jul 24 13:03:57.389878 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:57.389832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdlxq\" (UniqueName: \"kubernetes.io/projected/5c3202de-69a2-4a01-aa2a-3c3137db1ca5-kube-api-access-mdlxq\") pod \"payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz\" (UID: \"5c3202de-69a2-4a01-aa2a-3c3137db1ca5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz"
Jul 24 13:03:57.390050 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:57.390010    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c3202de-69a2-4a01-aa2a-3c3137db1ca5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz\" (UID: \"5c3202de-69a2-4a01-aa2a-3c3137db1ca5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz"
Jul 24 13:03:57.491586 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:57.491498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c3202de-69a2-4a01-aa2a-3c3137db1ca5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz\" (UID: \"5c3202de-69a2-4a01-aa2a-3c3137db1ca5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz"
Jul 24 13:03:57.491586 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:57.491553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdlxq\" (UniqueName: \"kubernetes.io/projected/5c3202de-69a2-4a01-aa2a-3c3137db1ca5-kube-api-access-mdlxq\") pod \"payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz\" (UID: \"5c3202de-69a2-4a01-aa2a-3c3137db1ca5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz"
Jul 24 13:03:57.492217 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:57.492196    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c3202de-69a2-4a01-aa2a-3c3137db1ca5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz\" (UID: \"5c3202de-69a2-4a01-aa2a-3c3137db1ca5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz"
Jul 24 13:03:57.500160 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:57.500129    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdlxq\" (UniqueName: \"kubernetes.io/projected/5c3202de-69a2-4a01-aa2a-3c3137db1ca5-kube-api-access-mdlxq\") pod \"payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz\" (UID: \"5c3202de-69a2-4a01-aa2a-3c3137db1ca5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz"
Jul 24 13:03:57.551059 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:57.551023    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz"
Jul 24 13:03:57.690434 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:57.690383    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz"]
Jul 24 13:03:57.693138 ip-10-0-128-194 kubenswrapper[2581]: W0724 13:03:57.693107    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3202de_69a2_4a01_aa2a_3c3137db1ca5.slice/crio-7e02549817784aa46c454f83a24d9bc00621519a1ac7a9880bea877d8521ffc8 WatchSource:0}: Error finding container 7e02549817784aa46c454f83a24d9bc00621519a1ac7a9880bea877d8521ffc8: Status 404 returned error can't find the container with id 7e02549817784aa46c454f83a24d9bc00621519a1ac7a9880bea877d8521ffc8
Jul 24 13:03:57.768805 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:57.768773    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz" event={"ID":"5c3202de-69a2-4a01-aa2a-3c3137db1ca5","Type":"ContainerStarted","Data":"6a12468fd5803efd76fdd84b42e439fa7036a39b9ad0bc251051bfb4fc3e6942"}
Jul 24 13:03:57.768984 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:57.768826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz" event={"ID":"5c3202de-69a2-4a01-aa2a-3c3137db1ca5","Type":"ContainerStarted","Data":"7e02549817784aa46c454f83a24d9bc00621519a1ac7a9880bea877d8521ffc8"}
Jul 24 13:03:57.768984 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:57.768867    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz"
Jul 24 13:03:57.785636 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:03:57.785581    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz" podStartSLOduration=0.785563948 podStartE2EDuration="785.563948ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:03:57.784367796 +0000 UTC m=+2669.739372577" watchObservedRunningTime="2026-07-24 13:03:57.785563948 +0000 UTC m=+2669.740568708"
Jul 24 13:04:04.397869 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:04.397819    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:04:08.776912 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:08.776877    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz"
Jul 24 13:04:16.469200 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:16.469164    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh"]
Jul 24 13:04:16.472860 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:16.472841    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh"
Jul 24 13:04:16.475120 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:16.475096    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-47a7ae\""
Jul 24 13:04:16.475488 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:16.475462    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-47a7ae-dockercfg-wkhbg\""
Jul 24 13:04:16.482313 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:16.482288    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh"]
Jul 24 13:04:16.568949 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:16.568912    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbfd566d-4728-427f-b16d-ac81ab780133-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh\" (UID: \"fbfd566d-4728-427f-b16d-ac81ab780133\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh"
Jul 24 13:04:16.569137 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:16.568971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b7rf\" (UniqueName: \"kubernetes.io/projected/fbfd566d-4728-427f-b16d-ac81ab780133-kube-api-access-2b7rf\") pod \"payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh\" (UID: \"fbfd566d-4728-427f-b16d-ac81ab780133\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh"
Jul 24 13:04:16.670403 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:16.670366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbfd566d-4728-427f-b16d-ac81ab780133-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh\" (UID: \"fbfd566d-4728-427f-b16d-ac81ab780133\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh"
Jul 24 13:04:16.670601 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:16.670476    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b7rf\" (UniqueName: \"kubernetes.io/projected/fbfd566d-4728-427f-b16d-ac81ab780133-kube-api-access-2b7rf\") pod \"payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh\" (UID: \"fbfd566d-4728-427f-b16d-ac81ab780133\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh"
Jul 24 13:04:16.671123 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:16.671105    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbfd566d-4728-427f-b16d-ac81ab780133-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh\" (UID: \"fbfd566d-4728-427f-b16d-ac81ab780133\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh"
Jul 24 13:04:16.679553 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:16.679521    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b7rf\" (UniqueName: \"kubernetes.io/projected/fbfd566d-4728-427f-b16d-ac81ab780133-kube-api-access-2b7rf\") pod \"payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh\" (UID: \"fbfd566d-4728-427f-b16d-ac81ab780133\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh"
Jul 24 13:04:16.784290 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:16.784254    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh"
Jul 24 13:04:16.921122 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:16.921097    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh"]
Jul 24 13:04:16.923279 ip-10-0-128-194 kubenswrapper[2581]: W0724 13:04:16.923250    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbfd566d_4728_427f_b16d_ac81ab780133.slice/crio-939178e5de38047df25b57b1fdc9e02cfe5ba48bf7db858f6f3c6bcc302e1534 WatchSource:0}: Error finding container 939178e5de38047df25b57b1fdc9e02cfe5ba48bf7db858f6f3c6bcc302e1534: Status 404 returned error can't find the container with id 939178e5de38047df25b57b1fdc9e02cfe5ba48bf7db858f6f3c6bcc302e1534
Jul 24 13:04:17.877027 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:17.876990    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh" event={"ID":"fbfd566d-4728-427f-b16d-ac81ab780133","Type":"ContainerStarted","Data":"9b6f7b5e7ad24c1326ddfd955757554fa51d98710140f8530dc2a4a7b3cc9107"}
Jul 24 13:04:17.877439 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:17.877046    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh" event={"ID":"fbfd566d-4728-427f-b16d-ac81ab780133","Type":"ContainerStarted","Data":"939178e5de38047df25b57b1fdc9e02cfe5ba48bf7db858f6f3c6bcc302e1534"}
Jul 24 13:04:17.877439 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:17.877066    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh"
Jul 24 13:04:17.898456 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:17.898388    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh" podStartSLOduration=1.898372133 podStartE2EDuration="1.898372133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:04:17.896773856 +0000 UTC m=+2689.851778622" watchObservedRunningTime="2026-07-24 13:04:17.898372133 +0000 UTC m=+2689.853376920"
Jul 24 13:04:19.200283 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:19.200203    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:04:28.884741 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:28.884701    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh"
Jul 24 13:04:48.726983 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.726950    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"]
Jul 24 13:04:48.730678 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.730660    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:48.733833 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.733808    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-967d3e\"/\"openshift-service-ca.crt\""
Jul 24 13:04:48.733980 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.733808    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-967d3e\"/\"default-dockercfg-v8ql4\""
Jul 24 13:04:48.733980 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.733808    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-967d3e\"/\"ipp-route-967d3e-kserve-self-signed-certs\""
Jul 24 13:04:48.733980 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.733811    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-967d3e\"/\"kube-root-ca.crt\""
Jul 24 13:04:48.740859 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.740836    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"]
Jul 24 13:04:48.878761 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.878724    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-dshm\") pod \"ipp-route-967d3e-kserve-778d56c95f-65ggw\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") " pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:48.878944 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.878772    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0441e976-0134-4db2-b157-c8bde4293c0a-tls-certs\") pod \"ipp-route-967d3e-kserve-778d56c95f-65ggw\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") " pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:48.878944 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.878879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds9rn\" (UniqueName: \"kubernetes.io/projected/0441e976-0134-4db2-b157-c8bde4293c0a-kube-api-access-ds9rn\") pod \"ipp-route-967d3e-kserve-778d56c95f-65ggw\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") " pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:48.878944 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.878913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-tmp-dir\") pod \"ipp-route-967d3e-kserve-778d56c95f-65ggw\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") " pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:48.879059 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.878987    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-home\") pod \"ipp-route-967d3e-kserve-778d56c95f-65ggw\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") " pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:48.879059 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.879029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-model-cache\") pod \"ipp-route-967d3e-kserve-778d56c95f-65ggw\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") " pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:48.980158 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.980066    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-home\") pod \"ipp-route-967d3e-kserve-778d56c95f-65ggw\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") " pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:48.980158 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.980127    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-model-cache\") pod \"ipp-route-967d3e-kserve-778d56c95f-65ggw\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") " pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:48.980368 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.980168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-dshm\") pod \"ipp-route-967d3e-kserve-778d56c95f-65ggw\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") " pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:48.980368 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.980210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0441e976-0134-4db2-b157-c8bde4293c0a-tls-certs\") pod \"ipp-route-967d3e-kserve-778d56c95f-65ggw\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") " pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:48.980368 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.980288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9rn\" (UniqueName: \"kubernetes.io/projected/0441e976-0134-4db2-b157-c8bde4293c0a-kube-api-access-ds9rn\") pod \"ipp-route-967d3e-kserve-778d56c95f-65ggw\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") " pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:48.980368 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.980317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-tmp-dir\") pod \"ipp-route-967d3e-kserve-778d56c95f-65ggw\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") " pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:48.980589 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.980573    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-home\") pod \"ipp-route-967d3e-kserve-778d56c95f-65ggw\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") " pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:48.980630 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.980608    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-model-cache\") pod \"ipp-route-967d3e-kserve-778d56c95f-65ggw\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") " pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:48.980670 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.980651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-tmp-dir\") pod \"ipp-route-967d3e-kserve-778d56c95f-65ggw\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") " pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:48.982657 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.982635    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-dshm\") pod \"ipp-route-967d3e-kserve-778d56c95f-65ggw\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") " pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:48.982971 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.982954    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0441e976-0134-4db2-b157-c8bde4293c0a-tls-certs\") pod \"ipp-route-967d3e-kserve-778d56c95f-65ggw\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") " pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:48.988139 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:48.988114    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds9rn\" (UniqueName: \"kubernetes.io/projected/0441e976-0134-4db2-b157-c8bde4293c0a-kube-api-access-ds9rn\") pod \"ipp-route-967d3e-kserve-778d56c95f-65ggw\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") " pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:49.041390 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:49.041356    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:49.179284 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:49.179256    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"]
Jul 24 13:04:49.181561 ip-10-0-128-194 kubenswrapper[2581]: W0724 13:04:49.181534    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0441e976_0134_4db2_b157_c8bde4293c0a.slice/crio-70c7d18963b1f922c9cfcfe3be0e0bceae2b0a0c284d8ef97394257ceb504344 WatchSource:0}: Error finding container 70c7d18963b1f922c9cfcfe3be0e0bceae2b0a0c284d8ef97394257ceb504344: Status 404 returned error can't find the container with id 70c7d18963b1f922c9cfcfe3be0e0bceae2b0a0c284d8ef97394257ceb504344
Jul 24 13:04:50.043397 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:50.043354    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw" event={"ID":"0441e976-0134-4db2-b157-c8bde4293c0a","Type":"ContainerStarted","Data":"c6564f18bf92bc0d22f726b7af58082cdb094d0b879e4f9714780565639c278a"}
Jul 24 13:04:50.043817 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:50.043409    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw" event={"ID":"0441e976-0134-4db2-b157-c8bde4293c0a","Type":"ContainerStarted","Data":"70c7d18963b1f922c9cfcfe3be0e0bceae2b0a0c284d8ef97394257ceb504344"}
Jul 24 13:04:50.061776 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:50.061722    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw" podStartSLOduration=2.061706599 podStartE2EDuration="2.061706599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:04:50.05974189 +0000 UTC m=+2722.014746653" watchObservedRunningTime="2026-07-24 13:04:50.061706599 +0000 UTC m=+2722.016711362"
Jul 24 13:04:52.501054 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:52.501016    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:04:59.042381 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:59.042338    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:59.042937 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:59.042394    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:59.055703 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:59.055677    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:04:59.100984 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:04:59.100954    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:05:17.801408 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:17.801371    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:05:33.935045 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:33.935007    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng"]
Jul 24 13:05:33.938838 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:33.938811    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng"
Jul 24 13:05:33.941677 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:33.941654    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-30955e-dockercfg-k422v\""
Jul 24 13:05:33.941972 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:33.941951    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-30955e\""
Jul 24 13:05:33.950641 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:33.950614    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng"]
Jul 24 13:05:33.962348 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:33.962319    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g"]
Jul 24 13:05:33.965991 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:33.965971    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g"
Jul 24 13:05:33.979261 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:33.979236    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g"]
Jul 24 13:05:34.110596 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:34.110557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-929v5\" (UniqueName: \"kubernetes.io/projected/e5081ed7-52fa-4ddd-82d0-0bb45ca71789-kube-api-access-929v5\") pod \"payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g\" (UID: \"e5081ed7-52fa-4ddd-82d0-0bb45ca71789\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g"
Jul 24 13:05:34.110772 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:34.110609    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5081ed7-52fa-4ddd-82d0-0bb45ca71789-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g\" (UID: \"e5081ed7-52fa-4ddd-82d0-0bb45ca71789\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g"
Jul 24 13:05:34.110772 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:34.110677    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69917387-c85e-46ed-8945-13cbd04a7529-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng\" (UID: \"69917387-c85e-46ed-8945-13cbd04a7529\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng"
Jul 24 13:05:34.110772 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:34.110707    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqmpp\" (UniqueName: \"kubernetes.io/projected/69917387-c85e-46ed-8945-13cbd04a7529-kube-api-access-rqmpp\") pod \"payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng\" (UID: \"69917387-c85e-46ed-8945-13cbd04a7529\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng"
Jul 24 13:05:34.212105 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:34.212010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-929v5\" (UniqueName: \"kubernetes.io/projected/e5081ed7-52fa-4ddd-82d0-0bb45ca71789-kube-api-access-929v5\") pod \"payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g\" (UID: \"e5081ed7-52fa-4ddd-82d0-0bb45ca71789\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g"
Jul 24 13:05:34.212105 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:34.212056    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5081ed7-52fa-4ddd-82d0-0bb45ca71789-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g\" (UID: \"e5081ed7-52fa-4ddd-82d0-0bb45ca71789\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g"
Jul 24 13:05:34.212105 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:34.212082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69917387-c85e-46ed-8945-13cbd04a7529-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng\" (UID: \"69917387-c85e-46ed-8945-13cbd04a7529\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng"
Jul 24 13:05:34.212105 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:34.212101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqmpp\" (UniqueName: \"kubernetes.io/projected/69917387-c85e-46ed-8945-13cbd04a7529-kube-api-access-rqmpp\") pod \"payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng\" (UID: \"69917387-c85e-46ed-8945-13cbd04a7529\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng"
Jul 24 13:05:34.212820 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:34.212801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69917387-c85e-46ed-8945-13cbd04a7529-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng\" (UID: \"69917387-c85e-46ed-8945-13cbd04a7529\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng"
Jul 24 13:05:34.212868 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:34.212831    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5081ed7-52fa-4ddd-82d0-0bb45ca71789-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g\" (UID: \"e5081ed7-52fa-4ddd-82d0-0bb45ca71789\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g"
Jul 24 13:05:34.219633 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:34.219612    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-929v5\" (UniqueName: \"kubernetes.io/projected/e5081ed7-52fa-4ddd-82d0-0bb45ca71789-kube-api-access-929v5\") pod \"payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g\" (UID: \"e5081ed7-52fa-4ddd-82d0-0bb45ca71789\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g"
Jul 24 13:05:34.219717 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:34.219693    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqmpp\" (UniqueName: \"kubernetes.io/projected/69917387-c85e-46ed-8945-13cbd04a7529-kube-api-access-rqmpp\") pod \"payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng\" (UID: \"69917387-c85e-46ed-8945-13cbd04a7529\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng"
Jul 24 13:05:34.250750 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:34.250715    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng"
Jul 24 13:05:34.278740 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:34.278708    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g"
Jul 24 13:05:34.404097 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:34.404048    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng"]
Jul 24 13:05:34.408605 ip-10-0-128-194 kubenswrapper[2581]: W0724 13:05:34.408571    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69917387_c85e_46ed_8945_13cbd04a7529.slice/crio-af03e545d2fa5d4cb937eed6bfef39e7ca9b401105529ef1f0e35ec8c4078bf6 WatchSource:0}: Error finding container af03e545d2fa5d4cb937eed6bfef39e7ca9b401105529ef1f0e35ec8c4078bf6: Status 404 returned error can't find the container with id af03e545d2fa5d4cb937eed6bfef39e7ca9b401105529ef1f0e35ec8c4078bf6
Jul 24 13:05:34.435867 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:34.435844    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g"]
Jul 24 13:05:34.441873 ip-10-0-128-194 kubenswrapper[2581]: W0724 13:05:34.441843    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5081ed7_52fa_4ddd_82d0_0bb45ca71789.slice/crio-d703766c9b3d63b4f82e53efa05d8af1501f067e923b501ecaf1705f6e1e624c WatchSource:0}: Error finding container d703766c9b3d63b4f82e53efa05d8af1501f067e923b501ecaf1705f6e1e624c: Status 404 returned error can't find the container with id d703766c9b3d63b4f82e53efa05d8af1501f067e923b501ecaf1705f6e1e624c
Jul 24 13:05:35.280939 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:35.280901    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng" event={"ID":"69917387-c85e-46ed-8945-13cbd04a7529","Type":"ContainerStarted","Data":"bd46653f9887e22885c9d82c9131907284823d2822fa44767e8f91dd256e555a"}
Jul 24 13:05:35.281395 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:35.280971    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng"
Jul 24 13:05:35.281395 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:35.280993    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng" event={"ID":"69917387-c85e-46ed-8945-13cbd04a7529","Type":"ContainerStarted","Data":"af03e545d2fa5d4cb937eed6bfef39e7ca9b401105529ef1f0e35ec8c4078bf6"}
Jul 24 13:05:35.282615 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:35.282587    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g" event={"ID":"e5081ed7-52fa-4ddd-82d0-0bb45ca71789","Type":"ContainerStarted","Data":"a90bce98b071c3d5a00d2d9609d2ab25cd815dc9e8ffa768211e0fbd6d7d97b5"}
Jul 24 13:05:35.282711 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:35.282636    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g" event={"ID":"e5081ed7-52fa-4ddd-82d0-0bb45ca71789","Type":"ContainerStarted","Data":"d703766c9b3d63b4f82e53efa05d8af1501f067e923b501ecaf1705f6e1e624c"}
Jul 24 13:05:35.282711 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:35.282659    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g"
Jul 24 13:05:35.297682 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:35.297635    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng" podStartSLOduration=2.297620398 podStartE2EDuration="2.297620398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:05:35.295274794 +0000 UTC m=+2767.250279558" watchObservedRunningTime="2026-07-24 13:05:35.297620398 +0000 UTC m=+2767.252625161"
Jul 24 13:05:35.310532 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:35.310481    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g" podStartSLOduration=2.31046491 podStartE2EDuration="2.31046491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:05:35.309134429 +0000 UTC m=+2767.264139217" watchObservedRunningTime="2026-07-24 13:05:35.31046491 +0000 UTC m=+2767.265469672"
Jul 24 13:05:38.101245 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:38.101215    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:05:46.289445 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:46.289359    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g" podUID="e5081ed7-52fa-4ddd-82d0-0bb45ca71789" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.60:9004: connect: connection refused"
Jul 24 13:05:46.290220 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:46.290195    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng"
Jul 24 13:05:56.288328 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:56.288298    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g"
Jul 24 13:05:58.637195 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:58.637160    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-30955e-nzrk6"]
Jul 24 13:05:58.642715 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:58.642692    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-30955e-nzrk6"
Jul 24 13:05:58.645194 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:58.645166    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zw8hg\""
Jul 24 13:05:58.645328 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:58.645214    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 13:05:58.651748 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:58.651720    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-30955e-nzrk6"]
Jul 24 13:05:58.750165 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:58.750119    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6885977a-45ed-4cae-be6a-beeb9c82d688-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-30955e-nzrk6\" (UID: \"6885977a-45ed-4cae-be6a-beeb9c82d688\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-30955e-nzrk6"
Jul 24 13:05:58.850689 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:58.850650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6885977a-45ed-4cae-be6a-beeb9c82d688-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-30955e-nzrk6\" (UID: \"6885977a-45ed-4cae-be6a-beeb9c82d688\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-30955e-nzrk6"
Jul 24 13:05:58.851329 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:58.851308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6885977a-45ed-4cae-be6a-beeb9c82d688-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-30955e-nzrk6\" (UID: \"6885977a-45ed-4cae-be6a-beeb9c82d688\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-30955e-nzrk6"
Jul 24 13:05:58.963815 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:58.963723    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-30955e-nzrk6"
Jul 24 13:05:59.104244 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:59.104214    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-30955e-nzrk6"]
Jul 24 13:05:59.105738 ip-10-0-128-194 kubenswrapper[2581]: W0724 13:05:59.105711    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6885977a_45ed_4cae_be6a_beeb9c82d688.slice/crio-0decb10d7d9d03dbb6f7e5fa4601eb60abeff140ee50f32006b1a03316699bbd WatchSource:0}: Error finding container 0decb10d7d9d03dbb6f7e5fa4601eb60abeff140ee50f32006b1a03316699bbd: Status 404 returned error can't find the container with id 0decb10d7d9d03dbb6f7e5fa4601eb60abeff140ee50f32006b1a03316699bbd
Jul 24 13:05:59.421361 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:05:59.421322    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-30955e-nzrk6" event={"ID":"6885977a-45ed-4cae-be6a-beeb9c82d688","Type":"ContainerStarted","Data":"0decb10d7d9d03dbb6f7e5fa4601eb60abeff140ee50f32006b1a03316699bbd"}
Jul 24 13:06:00.430528 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:00.430491    2581 generic.go:356] "Generic (PLEG): container finished" podID="6885977a-45ed-4cae-be6a-beeb9c82d688" containerID="5900c986996d807635f6f0dd7c92ff69215e0c53395f4cd84a57a2bc53eee936" exitCode=0
Jul 24 13:06:00.430947 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:00.430529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-30955e-nzrk6" event={"ID":"6885977a-45ed-4cae-be6a-beeb9c82d688","Type":"ContainerDied","Data":"5900c986996d807635f6f0dd7c92ff69215e0c53395f4cd84a57a2bc53eee936"}
Jul 24 13:06:01.580033 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:01.580003    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-30955e-nzrk6"
Jul 24 13:06:01.678800 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:01.678757    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6885977a-45ed-4cae-be6a-beeb9c82d688-maas-api-service-ca\") pod \"6885977a-45ed-4cae-be6a-beeb9c82d688\" (UID: \"6885977a-45ed-4cae-be6a-beeb9c82d688\") "
Jul 24 13:06:01.679146 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:01.679125    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6885977a-45ed-4cae-be6a-beeb9c82d688-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6885977a-45ed-4cae-be6a-beeb9c82d688" (UID: "6885977a-45ed-4cae-be6a-beeb9c82d688"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:06:01.780335 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:01.780238    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6885977a-45ed-4cae-be6a-beeb9c82d688-maas-api-service-ca\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:06:02.443887 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:02.443857    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-30955e-nzrk6" event={"ID":"6885977a-45ed-4cae-be6a-beeb9c82d688","Type":"ContainerDied","Data":"0decb10d7d9d03dbb6f7e5fa4601eb60abeff140ee50f32006b1a03316699bbd"}
Jul 24 13:06:02.443887 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:02.443893    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-30955e-nzrk6"
Jul 24 13:06:02.443887 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:02.443906    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0decb10d7d9d03dbb6f7e5fa4601eb60abeff140ee50f32006b1a03316699bbd"
Jul 24 13:06:09.151289 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.150939    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng"]
Jul 24 13:06:09.151932 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.151218    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng" podUID="69917387-c85e-46ed-8945-13cbd04a7529" containerName="payload-processing" containerID="cri-o://bd46653f9887e22885c9d82c9131907284823d2822fa44767e8f91dd256e555a" gracePeriod=30
Jul 24 13:06:09.160148 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.160109    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g"]
Jul 24 13:06:09.161185 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.161156    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g" podUID="e5081ed7-52fa-4ddd-82d0-0bb45ca71789" containerName="payload-pre-processing" containerID="cri-o://a90bce98b071c3d5a00d2d9609d2ab25cd815dc9e8ffa768211e0fbd6d7d97b5" gracePeriod=30
Jul 24 13:06:09.443722 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.443686    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng"
Jul 24 13:06:09.446995 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.446968    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g"
Jul 24 13:06:09.483927 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.483891    2581 generic.go:356] "Generic (PLEG): container finished" podID="e5081ed7-52fa-4ddd-82d0-0bb45ca71789" containerID="a90bce98b071c3d5a00d2d9609d2ab25cd815dc9e8ffa768211e0fbd6d7d97b5" exitCode=0
Jul 24 13:06:09.484111 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.483947    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g"
Jul 24 13:06:09.484111 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.483979    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g" event={"ID":"e5081ed7-52fa-4ddd-82d0-0bb45ca71789","Type":"ContainerDied","Data":"a90bce98b071c3d5a00d2d9609d2ab25cd815dc9e8ffa768211e0fbd6d7d97b5"}
Jul 24 13:06:09.484111 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.484036    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g" event={"ID":"e5081ed7-52fa-4ddd-82d0-0bb45ca71789","Type":"ContainerDied","Data":"d703766c9b3d63b4f82e53efa05d8af1501f067e923b501ecaf1705f6e1e624c"}
Jul 24 13:06:09.484111 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.484059    2581 scope.go:117] "RemoveContainer" containerID="a90bce98b071c3d5a00d2d9609d2ab25cd815dc9e8ffa768211e0fbd6d7d97b5"
Jul 24 13:06:09.485632 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.485606    2581 generic.go:356] "Generic (PLEG): container finished" podID="69917387-c85e-46ed-8945-13cbd04a7529" containerID="bd46653f9887e22885c9d82c9131907284823d2822fa44767e8f91dd256e555a" exitCode=0
Jul 24 13:06:09.485763 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.485659    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng"
Jul 24 13:06:09.485763 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.485686    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng" event={"ID":"69917387-c85e-46ed-8945-13cbd04a7529","Type":"ContainerDied","Data":"bd46653f9887e22885c9d82c9131907284823d2822fa44767e8f91dd256e555a"}
Jul 24 13:06:09.485763 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.485729    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng" event={"ID":"69917387-c85e-46ed-8945-13cbd04a7529","Type":"ContainerDied","Data":"af03e545d2fa5d4cb937eed6bfef39e7ca9b401105529ef1f0e35ec8c4078bf6"}
Jul 24 13:06:09.493637 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.493618    2581 scope.go:117] "RemoveContainer" containerID="a90bce98b071c3d5a00d2d9609d2ab25cd815dc9e8ffa768211e0fbd6d7d97b5"
Jul 24 13:06:09.493903 ip-10-0-128-194 kubenswrapper[2581]: E0724 13:06:09.493881    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90bce98b071c3d5a00d2d9609d2ab25cd815dc9e8ffa768211e0fbd6d7d97b5\": container with ID starting with a90bce98b071c3d5a00d2d9609d2ab25cd815dc9e8ffa768211e0fbd6d7d97b5 not found: ID does not exist" containerID="a90bce98b071c3d5a00d2d9609d2ab25cd815dc9e8ffa768211e0fbd6d7d97b5"
Jul 24 13:06:09.493978 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.493912    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90bce98b071c3d5a00d2d9609d2ab25cd815dc9e8ffa768211e0fbd6d7d97b5"} err="failed to get container status \"a90bce98b071c3d5a00d2d9609d2ab25cd815dc9e8ffa768211e0fbd6d7d97b5\": rpc error: code = NotFound desc = could not find container \"a90bce98b071c3d5a00d2d9609d2ab25cd815dc9e8ffa768211e0fbd6d7d97b5\": container with ID starting with a90bce98b071c3d5a00d2d9609d2ab25cd815dc9e8ffa768211e0fbd6d7d97b5 not found: ID does not exist"
Jul 24 13:06:09.493978 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.493929    2581 scope.go:117] "RemoveContainer" containerID="bd46653f9887e22885c9d82c9131907284823d2822fa44767e8f91dd256e555a"
Jul 24 13:06:09.502480 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.502463    2581 scope.go:117] "RemoveContainer" containerID="bd46653f9887e22885c9d82c9131907284823d2822fa44767e8f91dd256e555a"
Jul 24 13:06:09.502745 ip-10-0-128-194 kubenswrapper[2581]: E0724 13:06:09.502727    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd46653f9887e22885c9d82c9131907284823d2822fa44767e8f91dd256e555a\": container with ID starting with bd46653f9887e22885c9d82c9131907284823d2822fa44767e8f91dd256e555a not found: ID does not exist" containerID="bd46653f9887e22885c9d82c9131907284823d2822fa44767e8f91dd256e555a"
Jul 24 13:06:09.502814 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.502755    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd46653f9887e22885c9d82c9131907284823d2822fa44767e8f91dd256e555a"} err="failed to get container status \"bd46653f9887e22885c9d82c9131907284823d2822fa44767e8f91dd256e555a\": rpc error: code = NotFound desc = could not find container \"bd46653f9887e22885c9d82c9131907284823d2822fa44767e8f91dd256e555a\": container with ID starting with bd46653f9887e22885c9d82c9131907284823d2822fa44767e8f91dd256e555a not found: ID does not exist"
Jul 24 13:06:09.556148 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.556117    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69917387-c85e-46ed-8945-13cbd04a7529-plugins-config-volume\") pod \"69917387-c85e-46ed-8945-13cbd04a7529\" (UID: \"69917387-c85e-46ed-8945-13cbd04a7529\") "
Jul 24 13:06:09.556148 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.556159    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqmpp\" (UniqueName: \"kubernetes.io/projected/69917387-c85e-46ed-8945-13cbd04a7529-kube-api-access-rqmpp\") pod \"69917387-c85e-46ed-8945-13cbd04a7529\" (UID: \"69917387-c85e-46ed-8945-13cbd04a7529\") "
Jul 24 13:06:09.556398 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.556196    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-929v5\" (UniqueName: \"kubernetes.io/projected/e5081ed7-52fa-4ddd-82d0-0bb45ca71789-kube-api-access-929v5\") pod \"e5081ed7-52fa-4ddd-82d0-0bb45ca71789\" (UID: \"e5081ed7-52fa-4ddd-82d0-0bb45ca71789\") "
Jul 24 13:06:09.556398 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.556277    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5081ed7-52fa-4ddd-82d0-0bb45ca71789-plugins-config-volume\") pod \"e5081ed7-52fa-4ddd-82d0-0bb45ca71789\" (UID: \"e5081ed7-52fa-4ddd-82d0-0bb45ca71789\") "
Jul 24 13:06:09.556652 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.556624    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69917387-c85e-46ed-8945-13cbd04a7529-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69917387-c85e-46ed-8945-13cbd04a7529" (UID: "69917387-c85e-46ed-8945-13cbd04a7529"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:06:09.556722 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.556670    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5081ed7-52fa-4ddd-82d0-0bb45ca71789-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e5081ed7-52fa-4ddd-82d0-0bb45ca71789" (UID: "e5081ed7-52fa-4ddd-82d0-0bb45ca71789"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:06:09.558533 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.558511    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69917387-c85e-46ed-8945-13cbd04a7529-kube-api-access-rqmpp" (OuterVolumeSpecName: "kube-api-access-rqmpp") pod "69917387-c85e-46ed-8945-13cbd04a7529" (UID: "69917387-c85e-46ed-8945-13cbd04a7529"). InnerVolumeSpecName "kube-api-access-rqmpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:06:09.558613 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.558552    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5081ed7-52fa-4ddd-82d0-0bb45ca71789-kube-api-access-929v5" (OuterVolumeSpecName: "kube-api-access-929v5") pod "e5081ed7-52fa-4ddd-82d0-0bb45ca71789" (UID: "e5081ed7-52fa-4ddd-82d0-0bb45ca71789"). InnerVolumeSpecName "kube-api-access-929v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:06:09.657676 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.657632    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69917387-c85e-46ed-8945-13cbd04a7529-plugins-config-volume\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:06:09.657676 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.657667    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqmpp\" (UniqueName: \"kubernetes.io/projected/69917387-c85e-46ed-8945-13cbd04a7529-kube-api-access-rqmpp\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:06:09.657676 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.657676    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-929v5\" (UniqueName: \"kubernetes.io/projected/e5081ed7-52fa-4ddd-82d0-0bb45ca71789-kube-api-access-929v5\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:06:09.657676 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.657686    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5081ed7-52fa-4ddd-82d0-0bb45ca71789-plugins-config-volume\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:06:09.811911 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.811871    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng"]
Jul 24 13:06:09.817785 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.817760    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-30955e-54578dfd-xgcng"]
Jul 24 13:06:09.826278 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.826252    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g"]
Jul 24 13:06:09.830537 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:09.830512    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-30955e-fc895bc96-7n68g"]
Jul 24 13:06:10.647616 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:10.647582    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69917387-c85e-46ed-8945-13cbd04a7529" path="/var/lib/kubelet/pods/69917387-c85e-46ed-8945-13cbd04a7529/volumes"
Jul 24 13:06:10.648037 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:10.647953    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5081ed7-52fa-4ddd-82d0-0bb45ca71789" path="/var/lib/kubelet/pods/e5081ed7-52fa-4ddd-82d0-0bb45ca71789/volumes"
Jul 24 13:06:13.610628 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:13.610596    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:06:13.719253 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:13.719210    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-30955e-nzrk6"]
Jul 24 13:06:13.721295 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:13.721271    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-30955e-nzrk6"]
Jul 24 13:06:14.647374 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:14.647337    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6885977a-45ed-4cae-be6a-beeb9c82d688" path="/var/lib/kubelet/pods/6885977a-45ed-4cae-be6a-beeb9c82d688/volumes"
Jul 24 13:06:24.035707 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:24.035667    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-967d3e-6jfvf"]
Jul 24 13:06:24.036112 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:24.036088    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69917387-c85e-46ed-8945-13cbd04a7529" containerName="payload-processing"
Jul 24 13:06:24.036112 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:24.036099    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="69917387-c85e-46ed-8945-13cbd04a7529" containerName="payload-processing"
Jul 24 13:06:24.036112 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:24.036106    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5081ed7-52fa-4ddd-82d0-0bb45ca71789" containerName="payload-pre-processing"
Jul 24 13:06:24.036112 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:24.036111    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5081ed7-52fa-4ddd-82d0-0bb45ca71789" containerName="payload-pre-processing"
Jul 24 13:06:24.036244 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:24.036123    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6885977a-45ed-4cae-be6a-beeb9c82d688" containerName="revoke-keys"
Jul 24 13:06:24.036244 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:24.036129    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6885977a-45ed-4cae-be6a-beeb9c82d688" containerName="revoke-keys"
Jul 24 13:06:24.036244 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:24.036189    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="69917387-c85e-46ed-8945-13cbd04a7529" containerName="payload-processing"
Jul 24 13:06:24.036244 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:24.036200    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6885977a-45ed-4cae-be6a-beeb9c82d688" containerName="revoke-keys"
Jul 24 13:06:24.036244 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:24.036208    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5081ed7-52fa-4ddd-82d0-0bb45ca71789" containerName="payload-pre-processing"
Jul 24 13:06:24.040719 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:24.040701    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-967d3e-6jfvf"
Jul 24 13:06:24.042737 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:24.042714    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zw8hg\""
Jul 24 13:06:24.042857 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:24.042717    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 13:06:24.047781 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:24.047759    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-967d3e-6jfvf"]
Jul 24 13:06:24.092671 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:24.092630    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1079a5-9e02-4b06-b63f-920dee35fe06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-967d3e-6jfvf\" (UID: \"5b1079a5-9e02-4b06-b63f-920dee35fe06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-967d3e-6jfvf"
Jul 24 13:06:24.194215 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:24.194176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1079a5-9e02-4b06-b63f-920dee35fe06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-967d3e-6jfvf\" (UID: \"5b1079a5-9e02-4b06-b63f-920dee35fe06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-967d3e-6jfvf"
Jul 24 13:06:24.194831 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:24.194811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1079a5-9e02-4b06-b63f-920dee35fe06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-967d3e-6jfvf\" (UID: \"5b1079a5-9e02-4b06-b63f-920dee35fe06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-967d3e-6jfvf"
Jul 24 13:06:24.353551 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:24.353462    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-967d3e-6jfvf"
Jul 24 13:06:24.488615 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:24.488591    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-967d3e-6jfvf"]
Jul 24 13:06:24.490809 ip-10-0-128-194 kubenswrapper[2581]: W0724 13:06:24.490780    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1079a5_9e02_4b06_b63f_920dee35fe06.slice/crio-8e59ea0be182cc9513621ea293ada4a902bb3371158308d9cdfc5b8c5b4b51d4 WatchSource:0}: Error finding container 8e59ea0be182cc9513621ea293ada4a902bb3371158308d9cdfc5b8c5b4b51d4: Status 404 returned error can't find the container with id 8e59ea0be182cc9513621ea293ada4a902bb3371158308d9cdfc5b8c5b4b51d4
Jul 24 13:06:24.577511 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:24.577466    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-967d3e-6jfvf" event={"ID":"5b1079a5-9e02-4b06-b63f-920dee35fe06","Type":"ContainerStarted","Data":"8e59ea0be182cc9513621ea293ada4a902bb3371158308d9cdfc5b8c5b4b51d4"}
Jul 24 13:06:25.584833 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:25.584800    2581 generic.go:356] "Generic (PLEG): container finished" podID="5b1079a5-9e02-4b06-b63f-920dee35fe06" containerID="d4d4a0f05dc9e94443c470451c69d56a6c4c6fe209990a8f1b892bc88b62d4fd" exitCode=0
Jul 24 13:06:25.585217 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:25.584884    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-967d3e-6jfvf" event={"ID":"5b1079a5-9e02-4b06-b63f-920dee35fe06","Type":"ContainerDied","Data":"d4d4a0f05dc9e94443c470451c69d56a6c4c6fe209990a8f1b892bc88b62d4fd"}
Jul 24 13:06:26.734390 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:26.734363    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-967d3e-6jfvf"
Jul 24 13:06:26.819310 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:26.819277    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1079a5-9e02-4b06-b63f-920dee35fe06-maas-api-service-ca\") pod \"5b1079a5-9e02-4b06-b63f-920dee35fe06\" (UID: \"5b1079a5-9e02-4b06-b63f-920dee35fe06\") "
Jul 24 13:06:26.819631 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:26.819607    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b1079a5-9e02-4b06-b63f-920dee35fe06-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b1079a5-9e02-4b06-b63f-920dee35fe06" (UID: "5b1079a5-9e02-4b06-b63f-920dee35fe06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:06:26.920847 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:26.920754    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1079a5-9e02-4b06-b63f-920dee35fe06-maas-api-service-ca\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:06:27.596281 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:27.596246    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-967d3e-6jfvf"
Jul 24 13:06:27.596493 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:27.596283    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-967d3e-6jfvf" event={"ID":"5b1079a5-9e02-4b06-b63f-920dee35fe06","Type":"ContainerDied","Data":"8e59ea0be182cc9513621ea293ada4a902bb3371158308d9cdfc5b8c5b4b51d4"}
Jul 24 13:06:27.596493 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:27.596358    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e59ea0be182cc9513621ea293ada4a902bb3371158308d9cdfc5b8c5b4b51d4"
Jul 24 13:06:39.209956 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:39.209916    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:06:39.597990 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:39.597958    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz"]
Jul 24 13:06:39.598200 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:39.598164    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz" podUID="5c3202de-69a2-4a01-aa2a-3c3137db1ca5" containerName="payload-pre-processing" containerID="cri-o://6a12468fd5803efd76fdd84b42e439fa7036a39b9ad0bc251051bfb4fc3e6942" gracePeriod=30
Jul 24 13:06:39.862182 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:39.862115    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz"
Jul 24 13:06:39.945265 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:39.945226    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c3202de-69a2-4a01-aa2a-3c3137db1ca5-plugins-config-volume\") pod \"5c3202de-69a2-4a01-aa2a-3c3137db1ca5\" (UID: \"5c3202de-69a2-4a01-aa2a-3c3137db1ca5\") "
Jul 24 13:06:39.945478 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:39.945296    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdlxq\" (UniqueName: \"kubernetes.io/projected/5c3202de-69a2-4a01-aa2a-3c3137db1ca5-kube-api-access-mdlxq\") pod \"5c3202de-69a2-4a01-aa2a-3c3137db1ca5\" (UID: \"5c3202de-69a2-4a01-aa2a-3c3137db1ca5\") "
Jul 24 13:06:39.945709 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:39.945680    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c3202de-69a2-4a01-aa2a-3c3137db1ca5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c3202de-69a2-4a01-aa2a-3c3137db1ca5" (UID: "5c3202de-69a2-4a01-aa2a-3c3137db1ca5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:06:39.947597 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:39.947571    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3202de-69a2-4a01-aa2a-3c3137db1ca5-kube-api-access-mdlxq" (OuterVolumeSpecName: "kube-api-access-mdlxq") pod "5c3202de-69a2-4a01-aa2a-3c3137db1ca5" (UID: "5c3202de-69a2-4a01-aa2a-3c3137db1ca5"). InnerVolumeSpecName "kube-api-access-mdlxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:06:40.047089 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:40.047053    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c3202de-69a2-4a01-aa2a-3c3137db1ca5-plugins-config-volume\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:06:40.047089 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:40.047092    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdlxq\" (UniqueName: \"kubernetes.io/projected/5c3202de-69a2-4a01-aa2a-3c3137db1ca5-kube-api-access-mdlxq\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:06:40.672692 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:40.672658    2581 generic.go:356] "Generic (PLEG): container finished" podID="5c3202de-69a2-4a01-aa2a-3c3137db1ca5" containerID="6a12468fd5803efd76fdd84b42e439fa7036a39b9ad0bc251051bfb4fc3e6942" exitCode=0
Jul 24 13:06:40.673147 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:40.672702    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz"
Jul 24 13:06:40.673147 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:40.672750    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz" event={"ID":"5c3202de-69a2-4a01-aa2a-3c3137db1ca5","Type":"ContainerDied","Data":"6a12468fd5803efd76fdd84b42e439fa7036a39b9ad0bc251051bfb4fc3e6942"}
Jul 24 13:06:40.673147 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:40.672803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz" event={"ID":"5c3202de-69a2-4a01-aa2a-3c3137db1ca5","Type":"ContainerDied","Data":"7e02549817784aa46c454f83a24d9bc00621519a1ac7a9880bea877d8521ffc8"}
Jul 24 13:06:40.673147 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:40.672829    2581 scope.go:117] "RemoveContainer" containerID="6a12468fd5803efd76fdd84b42e439fa7036a39b9ad0bc251051bfb4fc3e6942"
Jul 24 13:06:40.681617 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:40.681600    2581 scope.go:117] "RemoveContainer" containerID="6a12468fd5803efd76fdd84b42e439fa7036a39b9ad0bc251051bfb4fc3e6942"
Jul 24 13:06:40.681875 ip-10-0-128-194 kubenswrapper[2581]: E0724 13:06:40.681853    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a12468fd5803efd76fdd84b42e439fa7036a39b9ad0bc251051bfb4fc3e6942\": container with ID starting with 6a12468fd5803efd76fdd84b42e439fa7036a39b9ad0bc251051bfb4fc3e6942 not found: ID does not exist" containerID="6a12468fd5803efd76fdd84b42e439fa7036a39b9ad0bc251051bfb4fc3e6942"
Jul 24 13:06:40.681945 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:40.681887    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a12468fd5803efd76fdd84b42e439fa7036a39b9ad0bc251051bfb4fc3e6942"} err="failed to get container status \"6a12468fd5803efd76fdd84b42e439fa7036a39b9ad0bc251051bfb4fc3e6942\": rpc error: code = NotFound desc = could not find container \"6a12468fd5803efd76fdd84b42e439fa7036a39b9ad0bc251051bfb4fc3e6942\": container with ID starting with 6a12468fd5803efd76fdd84b42e439fa7036a39b9ad0bc251051bfb4fc3e6942 not found: ID does not exist"
Jul 24 13:06:40.688014 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:40.687990    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz"]
Jul 24 13:06:40.689882 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:40.689863    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-967d3e-57f5968fc6-t4jmz"]
Jul 24 13:06:42.646345 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:42.646311    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c3202de-69a2-4a01-aa2a-3c3137db1ca5" path="/var/lib/kubelet/pods/5c3202de-69a2-4a01-aa2a-3c3137db1ca5/volumes"
Jul 24 13:06:44.124893 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:44.124856    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-967d3e-6jfvf"]
Jul 24 13:06:44.127562 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:44.127535    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-967d3e-6jfvf"]
Jul 24 13:06:44.648437 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:44.648391    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b1079a5-9e02-4b06-b63f-920dee35fe06" path="/var/lib/kubelet/pods/5b1079a5-9e02-4b06-b63f-920dee35fe06/volumes"
Jul 24 13:06:49.514518 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:49.514484    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"]
Jul 24 13:06:49.803340 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:06:49.803299    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:07:04.535129 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.535075    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw" podUID="0441e976-0134-4db2-b157-c8bde4293c0a" containerName="main" containerID="cri-o://c6564f18bf92bc0d22f726b7af58082cdb094d0b879e4f9714780565639c278a" gracePeriod=45
Jul 24 13:07:04.792199 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.792139    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:07:04.803669 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.803632    2581 generic.go:356] "Generic (PLEG): container finished" podID="0441e976-0134-4db2-b157-c8bde4293c0a" containerID="c6564f18bf92bc0d22f726b7af58082cdb094d0b879e4f9714780565639c278a" exitCode=0
Jul 24 13:07:04.803811 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.803730    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"
Jul 24 13:07:04.803811 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.803747    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw" event={"ID":"0441e976-0134-4db2-b157-c8bde4293c0a","Type":"ContainerDied","Data":"c6564f18bf92bc0d22f726b7af58082cdb094d0b879e4f9714780565639c278a"}
Jul 24 13:07:04.803811 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.803794    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw" event={"ID":"0441e976-0134-4db2-b157-c8bde4293c0a","Type":"ContainerDied","Data":"70c7d18963b1f922c9cfcfe3be0e0bceae2b0a0c284d8ef97394257ceb504344"}
Jul 24 13:07:04.803968 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.803820    2581 scope.go:117] "RemoveContainer" containerID="c6564f18bf92bc0d22f726b7af58082cdb094d0b879e4f9714780565639c278a"
Jul 24 13:07:04.814401 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.814377    2581 scope.go:117] "RemoveContainer" containerID="c6564f18bf92bc0d22f726b7af58082cdb094d0b879e4f9714780565639c278a"
Jul 24 13:07:04.814748 ip-10-0-128-194 kubenswrapper[2581]: E0724 13:07:04.814716    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6564f18bf92bc0d22f726b7af58082cdb094d0b879e4f9714780565639c278a\": container with ID starting with c6564f18bf92bc0d22f726b7af58082cdb094d0b879e4f9714780565639c278a not found: ID does not exist" containerID="c6564f18bf92bc0d22f726b7af58082cdb094d0b879e4f9714780565639c278a"
Jul 24 13:07:04.814887 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.814758    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6564f18bf92bc0d22f726b7af58082cdb094d0b879e4f9714780565639c278a"} err="failed to get container status \"c6564f18bf92bc0d22f726b7af58082cdb094d0b879e4f9714780565639c278a\": rpc error: code = NotFound desc = could not find container \"c6564f18bf92bc0d22f726b7af58082cdb094d0b879e4f9714780565639c278a\": container with ID starting with c6564f18bf92bc0d22f726b7af58082cdb094d0b879e4f9714780565639c278a not found: ID does not exist"
Jul 24 13:07:04.899364 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.899332    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-home\") pod \"0441e976-0134-4db2-b157-c8bde4293c0a\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") "
Jul 24 13:07:04.899552 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.899394    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-tmp-dir\") pod \"0441e976-0134-4db2-b157-c8bde4293c0a\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") "
Jul 24 13:07:04.899552 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.899484    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ds9rn\" (UniqueName: \"kubernetes.io/projected/0441e976-0134-4db2-b157-c8bde4293c0a-kube-api-access-ds9rn\") pod \"0441e976-0134-4db2-b157-c8bde4293c0a\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") "
Jul 24 13:07:04.899552 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.899501    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-model-cache\") pod \"0441e976-0134-4db2-b157-c8bde4293c0a\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") "
Jul 24 13:07:04.899552 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.899539    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-dshm\") pod \"0441e976-0134-4db2-b157-c8bde4293c0a\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") "
Jul 24 13:07:04.899778 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.899634    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0441e976-0134-4db2-b157-c8bde4293c0a" (UID: "0441e976-0134-4db2-b157-c8bde4293c0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:07:04.899778 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.899653    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-home" (OuterVolumeSpecName: "home") pod "0441e976-0134-4db2-b157-c8bde4293c0a" (UID: "0441e976-0134-4db2-b157-c8bde4293c0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:07:04.899778 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.899696    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0441e976-0134-4db2-b157-c8bde4293c0a-tls-certs\") pod \"0441e976-0134-4db2-b157-c8bde4293c0a\" (UID: \"0441e976-0134-4db2-b157-c8bde4293c0a\") "
Jul 24 13:07:04.899928 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.899774    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-model-cache" (OuterVolumeSpecName: "model-cache") pod "0441e976-0134-4db2-b157-c8bde4293c0a" (UID: "0441e976-0134-4db2-b157-c8bde4293c0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:07:04.900066 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.900044    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-tmp-dir\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:07:04.900066 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.900067    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-model-cache\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:07:04.900227 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.900079    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-home\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:07:04.901878 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.901856    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0441e976-0134-4db2-b157-c8bde4293c0a-kube-api-access-ds9rn" (OuterVolumeSpecName: "kube-api-access-ds9rn") pod "0441e976-0134-4db2-b157-c8bde4293c0a" (UID: "0441e976-0134-4db2-b157-c8bde4293c0a"). InnerVolumeSpecName "kube-api-access-ds9rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:07:04.901960 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.901886    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0441e976-0134-4db2-b157-c8bde4293c0a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0441e976-0134-4db2-b157-c8bde4293c0a" (UID: "0441e976-0134-4db2-b157-c8bde4293c0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:07:04.901960 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:04.901920    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-dshm" (OuterVolumeSpecName: "dshm") pod "0441e976-0134-4db2-b157-c8bde4293c0a" (UID: "0441e976-0134-4db2-b157-c8bde4293c0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:07:05.001043 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:05.001004    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0441e976-0134-4db2-b157-c8bde4293c0a-dshm\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:07:05.001043 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:05.001036    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0441e976-0134-4db2-b157-c8bde4293c0a-tls-certs\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:07:05.001043 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:05.001046    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds9rn\" (UniqueName: \"kubernetes.io/projected/0441e976-0134-4db2-b157-c8bde4293c0a-kube-api-access-ds9rn\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:07:05.124997 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:05.124965    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"]
Jul 24 13:07:05.129923 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:05.129899    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-967d3e/ipp-route-967d3e-kserve-778d56c95f-65ggw"]
Jul 24 13:07:06.646250 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:06.646213    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0441e976-0134-4db2-b157-c8bde4293c0a" path="/var/lib/kubelet/pods/0441e976-0134-4db2-b157-c8bde4293c0a/volumes"
Jul 24 13:07:38.998974 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:38.998937    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47a7ae-zxnrs"]
Jul 24 13:07:38.999474 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:38.999383    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c3202de-69a2-4a01-aa2a-3c3137db1ca5" containerName="payload-pre-processing"
Jul 24 13:07:38.999474 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:38.999396    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c3202de-69a2-4a01-aa2a-3c3137db1ca5" containerName="payload-pre-processing"
Jul 24 13:07:38.999474 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:38.999409    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0441e976-0134-4db2-b157-c8bde4293c0a" containerName="main"
Jul 24 13:07:38.999474 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:38.999429    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0441e976-0134-4db2-b157-c8bde4293c0a" containerName="main"
Jul 24 13:07:38.999474 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:38.999438    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b1079a5-9e02-4b06-b63f-920dee35fe06" containerName="revoke-keys"
Jul 24 13:07:38.999474 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:38.999444    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b1079a5-9e02-4b06-b63f-920dee35fe06" containerName="revoke-keys"
Jul 24 13:07:38.999669 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:38.999510    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c3202de-69a2-4a01-aa2a-3c3137db1ca5" containerName="payload-pre-processing"
Jul 24 13:07:38.999669 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:38.999519    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0441e976-0134-4db2-b157-c8bde4293c0a" containerName="main"
Jul 24 13:07:38.999669 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:38.999528    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b1079a5-9e02-4b06-b63f-920dee35fe06" containerName="revoke-keys"
Jul 24 13:07:39.003075 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:39.003058    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47a7ae-zxnrs"
Jul 24 13:07:39.005489 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:39.005461    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 13:07:39.005489 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:39.005486    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zw8hg\""
Jul 24 13:07:39.011194 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:39.011170    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47a7ae-zxnrs"]
Jul 24 13:07:39.031198 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:39.031168    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a90a8667-c233-4dae-9787-e00a7bb81d6b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-47a7ae-zxnrs\" (UID: \"a90a8667-c233-4dae-9787-e00a7bb81d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47a7ae-zxnrs"
Jul 24 13:07:39.131848 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:39.131811    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a90a8667-c233-4dae-9787-e00a7bb81d6b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-47a7ae-zxnrs\" (UID: \"a90a8667-c233-4dae-9787-e00a7bb81d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47a7ae-zxnrs"
Jul 24 13:07:39.132483 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:39.132463    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a90a8667-c233-4dae-9787-e00a7bb81d6b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-47a7ae-zxnrs\" (UID: \"a90a8667-c233-4dae-9787-e00a7bb81d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47a7ae-zxnrs"
Jul 24 13:07:39.314370 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:39.314336    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47a7ae-zxnrs"
Jul 24 13:07:39.455518 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:39.455310    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47a7ae-zxnrs"]
Jul 24 13:07:39.458121 ip-10-0-128-194 kubenswrapper[2581]: W0724 13:07:39.458093    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90a8667_c233_4dae_9787_e00a7bb81d6b.slice/crio-517d7bdd8be142626156f68c869e1e57a5c933bb00aca3e86ca646c0f271665b WatchSource:0}: Error finding container 517d7bdd8be142626156f68c869e1e57a5c933bb00aca3e86ca646c0f271665b: Status 404 returned error can't find the container with id 517d7bdd8be142626156f68c869e1e57a5c933bb00aca3e86ca646c0f271665b
Jul 24 13:07:39.992158 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:39.992068    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47a7ae-zxnrs" event={"ID":"a90a8667-c233-4dae-9787-e00a7bb81d6b","Type":"ContainerStarted","Data":"32fac02b353318efe01fbdfe3083de143d9a11aa6a5b4548bcf347a0124ea3db"}
Jul 24 13:07:39.992158 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:39.992122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47a7ae-zxnrs" event={"ID":"a90a8667-c233-4dae-9787-e00a7bb81d6b","Type":"ContainerStarted","Data":"517d7bdd8be142626156f68c869e1e57a5c933bb00aca3e86ca646c0f271665b"}
Jul 24 13:07:40.007729 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:40.007674    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47a7ae-zxnrs" podStartSLOduration=2.007652643 podStartE2EDuration="2.007652643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:07:40.006606511 +0000 UTC m=+2891.961622341" watchObservedRunningTime="2026-07-24 13:07:40.007652643 +0000 UTC m=+2891.962657406"
Jul 24 13:07:40.998827 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:40.998785    2581 generic.go:356] "Generic (PLEG): container finished" podID="a90a8667-c233-4dae-9787-e00a7bb81d6b" containerID="32fac02b353318efe01fbdfe3083de143d9a11aa6a5b4548bcf347a0124ea3db" exitCode=0
Jul 24 13:07:40.999016 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:40.998843    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47a7ae-zxnrs" event={"ID":"a90a8667-c233-4dae-9787-e00a7bb81d6b","Type":"ContainerDied","Data":"32fac02b353318efe01fbdfe3083de143d9a11aa6a5b4548bcf347a0124ea3db"}
Jul 24 13:07:42.141648 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:42.141623    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47a7ae-zxnrs"
Jul 24 13:07:42.159926 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:42.159900    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a90a8667-c233-4dae-9787-e00a7bb81d6b-maas-api-service-ca\") pod \"a90a8667-c233-4dae-9787-e00a7bb81d6b\" (UID: \"a90a8667-c233-4dae-9787-e00a7bb81d6b\") "
Jul 24 13:07:42.160275 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:42.160243    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a90a8667-c233-4dae-9787-e00a7bb81d6b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a90a8667-c233-4dae-9787-e00a7bb81d6b" (UID: "a90a8667-c233-4dae-9787-e00a7bb81d6b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:07:42.160447 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:42.160403    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a90a8667-c233-4dae-9787-e00a7bb81d6b-maas-api-service-ca\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:07:43.011343 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:43.011261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47a7ae-zxnrs" event={"ID":"a90a8667-c233-4dae-9787-e00a7bb81d6b","Type":"ContainerDied","Data":"517d7bdd8be142626156f68c869e1e57a5c933bb00aca3e86ca646c0f271665b"}
Jul 24 13:07:43.011343 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:43.011308    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="517d7bdd8be142626156f68c869e1e57a5c933bb00aca3e86ca646c0f271665b"
Jul 24 13:07:43.011343 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:43.011272    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47a7ae-zxnrs"
Jul 24 13:07:49.748959 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:49.748922    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh"]
Jul 24 13:07:49.749503 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:49.749168    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh" podUID="fbfd566d-4728-427f-b16d-ac81ab780133" containerName="payload-pre-processing" containerID="cri-o://9b6f7b5e7ad24c1326ddfd955757554fa51d98710140f8530dc2a4a7b3cc9107" gracePeriod=30
Jul 24 13:07:50.007666 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:50.007601    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh"
Jul 24 13:07:50.054828 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:50.054796    2581 generic.go:356] "Generic (PLEG): container finished" podID="fbfd566d-4728-427f-b16d-ac81ab780133" containerID="9b6f7b5e7ad24c1326ddfd955757554fa51d98710140f8530dc2a4a7b3cc9107" exitCode=0
Jul 24 13:07:50.055005 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:50.054844    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh"
Jul 24 13:07:50.055005 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:50.054892    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh" event={"ID":"fbfd566d-4728-427f-b16d-ac81ab780133","Type":"ContainerDied","Data":"9b6f7b5e7ad24c1326ddfd955757554fa51d98710140f8530dc2a4a7b3cc9107"}
Jul 24 13:07:50.055005 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:50.054944    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh" event={"ID":"fbfd566d-4728-427f-b16d-ac81ab780133","Type":"ContainerDied","Data":"939178e5de38047df25b57b1fdc9e02cfe5ba48bf7db858f6f3c6bcc302e1534"}
Jul 24 13:07:50.055005 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:50.054970    2581 scope.go:117] "RemoveContainer" containerID="9b6f7b5e7ad24c1326ddfd955757554fa51d98710140f8530dc2a4a7b3cc9107"
Jul 24 13:07:50.064782 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:50.064759    2581 scope.go:117] "RemoveContainer" containerID="9b6f7b5e7ad24c1326ddfd955757554fa51d98710140f8530dc2a4a7b3cc9107"
Jul 24 13:07:50.065087 ip-10-0-128-194 kubenswrapper[2581]: E0724 13:07:50.065069    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b6f7b5e7ad24c1326ddfd955757554fa51d98710140f8530dc2a4a7b3cc9107\": container with ID starting with 9b6f7b5e7ad24c1326ddfd955757554fa51d98710140f8530dc2a4a7b3cc9107 not found: ID does not exist" containerID="9b6f7b5e7ad24c1326ddfd955757554fa51d98710140f8530dc2a4a7b3cc9107"
Jul 24 13:07:50.065141 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:50.065097    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b6f7b5e7ad24c1326ddfd955757554fa51d98710140f8530dc2a4a7b3cc9107"} err="failed to get container status \"9b6f7b5e7ad24c1326ddfd955757554fa51d98710140f8530dc2a4a7b3cc9107\": rpc error: code = NotFound desc = could not find container \"9b6f7b5e7ad24c1326ddfd955757554fa51d98710140f8530dc2a4a7b3cc9107\": container with ID starting with 9b6f7b5e7ad24c1326ddfd955757554fa51d98710140f8530dc2a4a7b3cc9107 not found: ID does not exist"
Jul 24 13:07:50.136351 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:50.136317    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbfd566d-4728-427f-b16d-ac81ab780133-plugins-config-volume\") pod \"fbfd566d-4728-427f-b16d-ac81ab780133\" (UID: \"fbfd566d-4728-427f-b16d-ac81ab780133\") "
Jul 24 13:07:50.136567 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:50.136380    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b7rf\" (UniqueName: \"kubernetes.io/projected/fbfd566d-4728-427f-b16d-ac81ab780133-kube-api-access-2b7rf\") pod \"fbfd566d-4728-427f-b16d-ac81ab780133\" (UID: \"fbfd566d-4728-427f-b16d-ac81ab780133\") "
Jul 24 13:07:50.136754 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:50.136724    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbfd566d-4728-427f-b16d-ac81ab780133-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fbfd566d-4728-427f-b16d-ac81ab780133" (UID: "fbfd566d-4728-427f-b16d-ac81ab780133"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:07:50.138612 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:50.138590    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbfd566d-4728-427f-b16d-ac81ab780133-kube-api-access-2b7rf" (OuterVolumeSpecName: "kube-api-access-2b7rf") pod "fbfd566d-4728-427f-b16d-ac81ab780133" (UID: "fbfd566d-4728-427f-b16d-ac81ab780133"). InnerVolumeSpecName "kube-api-access-2b7rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:07:50.237275 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:50.237236    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbfd566d-4728-427f-b16d-ac81ab780133-plugins-config-volume\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:07:50.237275 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:50.237270    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b7rf\" (UniqueName: \"kubernetes.io/projected/fbfd566d-4728-427f-b16d-ac81ab780133-kube-api-access-2b7rf\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:07:50.377634 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:50.377599    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh"]
Jul 24 13:07:50.382054 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:50.382027    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-47a7ae-56769b459b-nfdmh"]
Jul 24 13:07:50.647904 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:50.647814    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbfd566d-4728-427f-b16d-ac81ab780133" path="/var/lib/kubelet/pods/fbfd566d-4728-427f-b16d-ac81ab780133/volumes"
Jul 24 13:07:53.017673 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:53.017637    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:07:54.090284 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:54.090248    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47a7ae-zxnrs"]
Jul 24 13:07:54.093126 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:54.093099    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-47a7ae-zxnrs"]
Jul 24 13:07:54.645680 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:07:54.645648    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a90a8667-c233-4dae-9787-e00a7bb81d6b" path="/var/lib/kubelet/pods/a90a8667-c233-4dae-9787-e00a7bb81d6b/volumes"
Jul 24 13:08:01.038982 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:01.038952    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d15f64-p4t9r"]
Jul 24 13:08:01.039380 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:01.039351    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbfd566d-4728-427f-b16d-ac81ab780133" containerName="payload-pre-processing"
Jul 24 13:08:01.039380 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:01.039361    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbfd566d-4728-427f-b16d-ac81ab780133" containerName="payload-pre-processing"
Jul 24 13:08:01.039380 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:01.039373    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a90a8667-c233-4dae-9787-e00a7bb81d6b" containerName="revoke-keys"
Jul 24 13:08:01.039380 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:01.039379    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a90a8667-c233-4dae-9787-e00a7bb81d6b" containerName="revoke-keys"
Jul 24 13:08:01.039560 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:01.039457    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a90a8667-c233-4dae-9787-e00a7bb81d6b" containerName="revoke-keys"
Jul 24 13:08:01.039560 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:01.039468    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbfd566d-4728-427f-b16d-ac81ab780133" containerName="payload-pre-processing"
Jul 24 13:08:01.043732 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:01.043713    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d15f64-p4t9r"
Jul 24 13:08:01.048226 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:01.048200    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zw8hg\""
Jul 24 13:08:01.048369 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:01.048201    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 13:08:01.051783 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:01.051757    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d15f64-p4t9r"]
Jul 24 13:08:01.138072 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:01.138035    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3d79c2-6c46-49fc-9960-775005800a34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d15f64-p4t9r\" (UID: \"6d3d79c2-6c46-49fc-9960-775005800a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d15f64-p4t9r"
Jul 24 13:08:01.239016 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:01.238979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3d79c2-6c46-49fc-9960-775005800a34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d15f64-p4t9r\" (UID: \"6d3d79c2-6c46-49fc-9960-775005800a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d15f64-p4t9r"
Jul 24 13:08:01.239668 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:01.239648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3d79c2-6c46-49fc-9960-775005800a34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d15f64-p4t9r\" (UID: \"6d3d79c2-6c46-49fc-9960-775005800a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d15f64-p4t9r"
Jul 24 13:08:01.357270 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:01.357182    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d15f64-p4t9r"
Jul 24 13:08:01.496626 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:01.496604    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d15f64-p4t9r"]
Jul 24 13:08:01.498911 ip-10-0-128-194 kubenswrapper[2581]: W0724 13:08:01.498877    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3d79c2_6c46_49fc_9960_775005800a34.slice/crio-e59a39ca008d75784ef1e946e1da4062a678c0868daccc38d3e850f5af9ec05a WatchSource:0}: Error finding container e59a39ca008d75784ef1e946e1da4062a678c0868daccc38d3e850f5af9ec05a: Status 404 returned error can't find the container with id e59a39ca008d75784ef1e946e1da4062a678c0868daccc38d3e850f5af9ec05a
Jul 24 13:08:02.121129 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:02.121095    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d15f64-p4t9r" event={"ID":"6d3d79c2-6c46-49fc-9960-775005800a34","Type":"ContainerStarted","Data":"1527b4e765c16c6fa325a1c36c9f1ba023843e945e28f2dd0cb869b7e1c73db4"}
Jul 24 13:08:02.121564 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:02.121147    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d15f64-p4t9r" event={"ID":"6d3d79c2-6c46-49fc-9960-775005800a34","Type":"ContainerStarted","Data":"e59a39ca008d75784ef1e946e1da4062a678c0868daccc38d3e850f5af9ec05a"}
Jul 24 13:08:02.136578 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:02.136524    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d15f64-p4t9r" podStartSLOduration=1.136508512 podStartE2EDuration="1.136508512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:08:02.135132413 +0000 UTC m=+2914.090137175" watchObservedRunningTime="2026-07-24 13:08:02.136508512 +0000 UTC m=+2914.091513536"
Jul 24 13:08:03.127990 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:03.127955    2581 generic.go:356] "Generic (PLEG): container finished" podID="6d3d79c2-6c46-49fc-9960-775005800a34" containerID="1527b4e765c16c6fa325a1c36c9f1ba023843e945e28f2dd0cb869b7e1c73db4" exitCode=0
Jul 24 13:08:03.127990 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:03.127991    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d15f64-p4t9r" event={"ID":"6d3d79c2-6c46-49fc-9960-775005800a34","Type":"ContainerDied","Data":"1527b4e765c16c6fa325a1c36c9f1ba023843e945e28f2dd0cb869b7e1c73db4"}
Jul 24 13:08:04.270603 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:04.270576    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d15f64-p4t9r"
Jul 24 13:08:04.370353 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:04.370316    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3d79c2-6c46-49fc-9960-775005800a34-maas-api-service-ca\") pod \"6d3d79c2-6c46-49fc-9960-775005800a34\" (UID: \"6d3d79c2-6c46-49fc-9960-775005800a34\") "
Jul 24 13:08:04.370702 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:04.370676    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3d79c2-6c46-49fc-9960-775005800a34-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d3d79c2-6c46-49fc-9960-775005800a34" (UID: "6d3d79c2-6c46-49fc-9960-775005800a34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:08:04.471899 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:04.471821    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3d79c2-6c46-49fc-9960-775005800a34-maas-api-service-ca\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:08:05.140251 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:05.140221    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d15f64-p4t9r"
Jul 24 13:08:05.140448 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:05.140257    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d15f64-p4t9r" event={"ID":"6d3d79c2-6c46-49fc-9960-775005800a34","Type":"ContainerDied","Data":"e59a39ca008d75784ef1e946e1da4062a678c0868daccc38d3e850f5af9ec05a"}
Jul 24 13:08:05.140448 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:05.140298    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e59a39ca008d75784ef1e946e1da4062a678c0868daccc38d3e850f5af9ec05a"
Jul 24 13:08:11.995994 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:11.995954    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d15f64-29748300-66vdt"]
Jul 24 13:08:11.999107 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:11.999081    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d15f64-29748300-66vdt"]
Jul 24 13:08:12.033087 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.033055    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh"]
Jul 24 13:08:12.034832 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.033656    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh" podUID="aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2" containerName="payload-processing" containerID="cri-o://8201d26c801334a33b8bf6ee7ffbb2a292037e1c1b6d018266b5d67cacac0e9a" gracePeriod=30
Jul 24 13:08:12.043079 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.043049    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc"]
Jul 24 13:08:12.045846 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.045270    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc" podUID="61290433-cc74-4be0-ad06-6d2fe7d65cb6" containerName="payload-pre-processing" containerID="cri-o://97fb404b6a6027977ea3b318fc4ae36310e4a8954e4cf8621411382176e3f96d" gracePeriod=30
Jul 24 13:08:12.179475 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.179440    2581 generic.go:356] "Generic (PLEG): container finished" podID="aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2" containerID="8201d26c801334a33b8bf6ee7ffbb2a292037e1c1b6d018266b5d67cacac0e9a" exitCode=0
Jul 24 13:08:12.179641 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.179450    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh" event={"ID":"aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2","Type":"ContainerDied","Data":"8201d26c801334a33b8bf6ee7ffbb2a292037e1c1b6d018266b5d67cacac0e9a"}
Jul 24 13:08:12.181538 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.181514    2581 generic.go:356] "Generic (PLEG): container finished" podID="61290433-cc74-4be0-ad06-6d2fe7d65cb6" containerID="97fb404b6a6027977ea3b318fc4ae36310e4a8954e4cf8621411382176e3f96d" exitCode=0
Jul 24 13:08:12.181660 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.181584    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc" event={"ID":"61290433-cc74-4be0-ad06-6d2fe7d65cb6","Type":"ContainerDied","Data":"97fb404b6a6027977ea3b318fc4ae36310e4a8954e4cf8621411382176e3f96d"}
Jul 24 13:08:12.339580 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.339553    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc"
Jul 24 13:08:12.343055 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.343034    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh"
Jul 24 13:08:12.449497 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.449460    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsz68\" (UniqueName: \"kubernetes.io/projected/aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2-kube-api-access-nsz68\") pod \"aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2\" (UID: \"aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2\") "
Jul 24 13:08:12.449679 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.449583    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2-plugins-config-volume\") pod \"aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2\" (UID: \"aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2\") "
Jul 24 13:08:12.449679 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.449623    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61290433-cc74-4be0-ad06-6d2fe7d65cb6-plugins-config-volume\") pod \"61290433-cc74-4be0-ad06-6d2fe7d65cb6\" (UID: \"61290433-cc74-4be0-ad06-6d2fe7d65cb6\") "
Jul 24 13:08:12.449679 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.449668    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsqz2\" (UniqueName: \"kubernetes.io/projected/61290433-cc74-4be0-ad06-6d2fe7d65cb6-kube-api-access-wsqz2\") pod \"61290433-cc74-4be0-ad06-6d2fe7d65cb6\" (UID: \"61290433-cc74-4be0-ad06-6d2fe7d65cb6\") "
Jul 24 13:08:12.450027 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.450000    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61290433-cc74-4be0-ad06-6d2fe7d65cb6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "61290433-cc74-4be0-ad06-6d2fe7d65cb6" (UID: "61290433-cc74-4be0-ad06-6d2fe7d65cb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:08:12.450126 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.449998    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2" (UID: "aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:08:12.451781 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.451762    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2-kube-api-access-nsz68" (OuterVolumeSpecName: "kube-api-access-nsz68") pod "aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2" (UID: "aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2"). InnerVolumeSpecName "kube-api-access-nsz68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:08:12.451929 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.451907    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61290433-cc74-4be0-ad06-6d2fe7d65cb6-kube-api-access-wsqz2" (OuterVolumeSpecName: "kube-api-access-wsqz2") pod "61290433-cc74-4be0-ad06-6d2fe7d65cb6" (UID: "61290433-cc74-4be0-ad06-6d2fe7d65cb6"). InnerVolumeSpecName "kube-api-access-wsqz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:08:12.551256 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.551222    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2-plugins-config-volume\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:08:12.551256 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.551252    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61290433-cc74-4be0-ad06-6d2fe7d65cb6-plugins-config-volume\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:08:12.551485 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.551266    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsqz2\" (UniqueName: \"kubernetes.io/projected/61290433-cc74-4be0-ad06-6d2fe7d65cb6-kube-api-access-wsqz2\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:08:12.551485 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.551278    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsz68\" (UniqueName: \"kubernetes.io/projected/aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2-kube-api-access-nsz68\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:08:12.646575 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:12.646541    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0b505b3-3c99-446c-80cf-73e2bb3bdcfd" path="/var/lib/kubelet/pods/f0b505b3-3c99-446c-80cf-73e2bb3bdcfd/volumes"
Jul 24 13:08:13.187703 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:13.187669    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh"
Jul 24 13:08:13.188172 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:13.187702    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh" event={"ID":"aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2","Type":"ContainerDied","Data":"9d8b40c7ad31c602e01edd7325544af998caf7475c2a472510a19bba6f4a0cd3"}
Jul 24 13:08:13.188172 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:13.187775    2581 scope.go:117] "RemoveContainer" containerID="8201d26c801334a33b8bf6ee7ffbb2a292037e1c1b6d018266b5d67cacac0e9a"
Jul 24 13:08:13.189138 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:13.189029    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc" event={"ID":"61290433-cc74-4be0-ad06-6d2fe7d65cb6","Type":"ContainerDied","Data":"9dd7de6bcb12f1453facb5b4ac21cc0ab808e588767299983b16037e9cae9647"}
Jul 24 13:08:13.189138 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:13.189039    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc"
Jul 24 13:08:13.197031 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:13.197006    2581 scope.go:117] "RemoveContainer" containerID="97fb404b6a6027977ea3b318fc4ae36310e4a8954e4cf8621411382176e3f96d"
Jul 24 13:08:13.205624 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:13.205604    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc"]
Jul 24 13:08:13.209398 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:13.209378    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-d15f64-785b685688-s8hgc"]
Jul 24 13:08:13.218424 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:13.218390    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh"]
Jul 24 13:08:13.222338 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:13.222314    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-d15f64-857b9f8c98-s6kfh"]
Jul 24 13:08:14.645908 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:14.645868    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61290433-cc74-4be0-ad06-6d2fe7d65cb6" path="/var/lib/kubelet/pods/61290433-cc74-4be0-ad06-6d2fe7d65cb6/volumes"
Jul 24 13:08:14.646434 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:14.646226    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2" path="/var/lib/kubelet/pods/aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2/volumes"
Jul 24 13:08:15.313845 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:15.313812    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:08:16.144157 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:16.144124    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d15f64-p4t9r"]
Jul 24 13:08:16.147679 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:16.147656    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d15f64-p4t9r"]
Jul 24 13:08:16.646054 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:16.646021    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d3d79c2-6c46-49fc-9960-775005800a34" path="/var/lib/kubelet/pods/6d3d79c2-6c46-49fc-9960-775005800a34/volumes"
Jul 24 13:08:23.423016 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:23.422980    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6697a-8bpg5"]
Jul 24 13:08:23.423410 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:23.423353    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61290433-cc74-4be0-ad06-6d2fe7d65cb6" containerName="payload-pre-processing"
Jul 24 13:08:23.423410 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:23.423366    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="61290433-cc74-4be0-ad06-6d2fe7d65cb6" containerName="payload-pre-processing"
Jul 24 13:08:23.423410 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:23.423375    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2" containerName="payload-processing"
Jul 24 13:08:23.423410 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:23.423381    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2" containerName="payload-processing"
Jul 24 13:08:23.423410 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:23.423397    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d3d79c2-6c46-49fc-9960-775005800a34" containerName="revoke-keys"
Jul 24 13:08:23.423410 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:23.423403    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d3d79c2-6c46-49fc-9960-775005800a34" containerName="revoke-keys"
Jul 24 13:08:23.423652 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:23.423486    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d3d79c2-6c46-49fc-9960-775005800a34" containerName="revoke-keys"
Jul 24 13:08:23.423652 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:23.423496    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa4bdcdb-418b-4b8d-b4a5-0cbe53bbcfa2" containerName="payload-processing"
Jul 24 13:08:23.423652 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:23.423503    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="61290433-cc74-4be0-ad06-6d2fe7d65cb6" containerName="payload-pre-processing"
Jul 24 13:08:23.427880 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:23.427864    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6697a-8bpg5"
Jul 24 13:08:23.429897 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:23.429874    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 13:08:23.430078 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:23.430065    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zw8hg\""
Jul 24 13:08:23.433364 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:23.433344    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6697a-8bpg5"]
Jul 24 13:08:23.447203 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:23.447169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cfc9441-8830-4c62-a4e2-923604f514f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c6697a-8bpg5\" (UID: \"7cfc9441-8830-4c62-a4e2-923604f514f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6697a-8bpg5"
Jul 24 13:08:23.548557 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:23.548521    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cfc9441-8830-4c62-a4e2-923604f514f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c6697a-8bpg5\" (UID: \"7cfc9441-8830-4c62-a4e2-923604f514f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6697a-8bpg5"
Jul 24 13:08:23.549143 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:23.549125    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cfc9441-8830-4c62-a4e2-923604f514f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c6697a-8bpg5\" (UID: \"7cfc9441-8830-4c62-a4e2-923604f514f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6697a-8bpg5"
Jul 24 13:08:23.739526 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:23.739398    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6697a-8bpg5"
Jul 24 13:08:23.875965 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:23.875940    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6697a-8bpg5"]
Jul 24 13:08:23.878088 ip-10-0-128-194 kubenswrapper[2581]: W0724 13:08:23.878060    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cfc9441_8830_4c62_a4e2_923604f514f9.slice/crio-abeeed222e565a1399ffdd71f9dd67935f6e253bd2a240fb324bf29bc5625ddf WatchSource:0}: Error finding container abeeed222e565a1399ffdd71f9dd67935f6e253bd2a240fb324bf29bc5625ddf: Status 404 returned error can't find the container with id abeeed222e565a1399ffdd71f9dd67935f6e253bd2a240fb324bf29bc5625ddf
Jul 24 13:08:24.255411 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:24.255366    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6697a-8bpg5" event={"ID":"7cfc9441-8830-4c62-a4e2-923604f514f9","Type":"ContainerStarted","Data":"cf4d5ad442be89118498ff77fb9726d567e3d75f89ef58851a74ff65a64b5b7f"}
Jul 24 13:08:24.255617 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:24.255456    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6697a-8bpg5" event={"ID":"7cfc9441-8830-4c62-a4e2-923604f514f9","Type":"ContainerStarted","Data":"abeeed222e565a1399ffdd71f9dd67935f6e253bd2a240fb324bf29bc5625ddf"}
Jul 24 13:08:24.273820 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:24.273769    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6697a-8bpg5" podStartSLOduration=1.273754322 podStartE2EDuration="1.273754322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:08:24.270991123 +0000 UTC m=+2936.225995904" watchObservedRunningTime="2026-07-24 13:08:24.273754322 +0000 UTC m=+2936.228759142"
Jul 24 13:08:25.261833 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:25.261798    2581 generic.go:356] "Generic (PLEG): container finished" podID="7cfc9441-8830-4c62-a4e2-923604f514f9" containerID="cf4d5ad442be89118498ff77fb9726d567e3d75f89ef58851a74ff65a64b5b7f" exitCode=0
Jul 24 13:08:25.262205 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:25.261865    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6697a-8bpg5" event={"ID":"7cfc9441-8830-4c62-a4e2-923604f514f9","Type":"ContainerDied","Data":"cf4d5ad442be89118498ff77fb9726d567e3d75f89ef58851a74ff65a64b5b7f"}
Jul 24 13:08:26.400179 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:26.400150    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6697a-8bpg5"
Jul 24 13:08:26.473656 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:26.473622    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cfc9441-8830-4c62-a4e2-923604f514f9-maas-api-service-ca\") pod \"7cfc9441-8830-4c62-a4e2-923604f514f9\" (UID: \"7cfc9441-8830-4c62-a4e2-923604f514f9\") "
Jul 24 13:08:26.474012 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:26.473985    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cfc9441-8830-4c62-a4e2-923604f514f9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7cfc9441-8830-4c62-a4e2-923604f514f9" (UID: "7cfc9441-8830-4c62-a4e2-923604f514f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:08:26.574832 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:26.574797    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cfc9441-8830-4c62-a4e2-923604f514f9-maas-api-service-ca\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:08:27.274467 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:27.274408    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6697a-8bpg5" event={"ID":"7cfc9441-8830-4c62-a4e2-923604f514f9","Type":"ContainerDied","Data":"abeeed222e565a1399ffdd71f9dd67935f6e253bd2a240fb324bf29bc5625ddf"}
Jul 24 13:08:27.274653 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:27.274490    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abeeed222e565a1399ffdd71f9dd67935f6e253bd2a240fb324bf29bc5625ddf"
Jul 24 13:08:27.274653 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:27.274443    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6697a-8bpg5"
Jul 24 13:08:28.988843 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:28.988811    2581 scope.go:117] "RemoveContainer" containerID="4c741f4bc6b8f74190054a4003460bbbe9c411ebf44d755d4093bea0cf8bc78b"
Jul 24 13:08:34.019053 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.019019    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c6697a-29748300-gc72n"]
Jul 24 13:08:34.021682 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.021659    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c6697a-29748300-gc72n"]
Jul 24 13:08:34.049626 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.049593    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt"]
Jul 24 13:08:34.049857 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.049831    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt" podUID="1efabaab-7c52-45a8-88ab-aa701f749d89" containerName="payload-processing" containerID="cri-o://9d81f3f795316c75b92a0e1348df9128b6af661a427790939f5d6d14beb43062" gracePeriod=30
Jul 24 13:08:34.055529 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.055153    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8"]
Jul 24 13:08:34.056141 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.055984    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8" podUID="5bdf7e41-145a-4816-a198-3037a4552393" containerName="payload-pre-processing" containerID="cri-o://c1b914b343b8bc144239fb67bab6044985a901ebd5eaadb50ee75dfba1dcb10d" gracePeriod=30
Jul 24 13:08:34.323148 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.323115    2581 generic.go:356] "Generic (PLEG): container finished" podID="1efabaab-7c52-45a8-88ab-aa701f749d89" containerID="9d81f3f795316c75b92a0e1348df9128b6af661a427790939f5d6d14beb43062" exitCode=0
Jul 24 13:08:34.323308 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.323147    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt" event={"ID":"1efabaab-7c52-45a8-88ab-aa701f749d89","Type":"ContainerDied","Data":"9d81f3f795316c75b92a0e1348df9128b6af661a427790939f5d6d14beb43062"}
Jul 24 13:08:34.324920 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.324901    2581 generic.go:356] "Generic (PLEG): container finished" podID="5bdf7e41-145a-4816-a198-3037a4552393" containerID="c1b914b343b8bc144239fb67bab6044985a901ebd5eaadb50ee75dfba1dcb10d" exitCode=0
Jul 24 13:08:34.325010 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.324970    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8" event={"ID":"5bdf7e41-145a-4816-a198-3037a4552393","Type":"ContainerDied","Data":"c1b914b343b8bc144239fb67bab6044985a901ebd5eaadb50ee75dfba1dcb10d"}
Jul 24 13:08:34.344930 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.344905    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt"
Jul 24 13:08:34.348161 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.348138    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8"
Jul 24 13:08:34.443226 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.443193    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1efabaab-7c52-45a8-88ab-aa701f749d89-plugins-config-volume\") pod \"1efabaab-7c52-45a8-88ab-aa701f749d89\" (UID: \"1efabaab-7c52-45a8-88ab-aa701f749d89\") "
Jul 24 13:08:34.443226 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.443232    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j52mp\" (UniqueName: \"kubernetes.io/projected/1efabaab-7c52-45a8-88ab-aa701f749d89-kube-api-access-j52mp\") pod \"1efabaab-7c52-45a8-88ab-aa701f749d89\" (UID: \"1efabaab-7c52-45a8-88ab-aa701f749d89\") "
Jul 24 13:08:34.443539 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.443294    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bdf7e41-145a-4816-a198-3037a4552393-plugins-config-volume\") pod \"5bdf7e41-145a-4816-a198-3037a4552393\" (UID: \"5bdf7e41-145a-4816-a198-3037a4552393\") "
Jul 24 13:08:34.443539 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.443359    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbm6z\" (UniqueName: \"kubernetes.io/projected/5bdf7e41-145a-4816-a198-3037a4552393-kube-api-access-fbm6z\") pod \"5bdf7e41-145a-4816-a198-3037a4552393\" (UID: \"5bdf7e41-145a-4816-a198-3037a4552393\") "
Jul 24 13:08:34.443710 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.443687    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1efabaab-7c52-45a8-88ab-aa701f749d89-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1efabaab-7c52-45a8-88ab-aa701f749d89" (UID: "1efabaab-7c52-45a8-88ab-aa701f749d89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:08:34.443779 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.443756    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bdf7e41-145a-4816-a198-3037a4552393-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5bdf7e41-145a-4816-a198-3037a4552393" (UID: "5bdf7e41-145a-4816-a198-3037a4552393"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:08:34.445652 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.445629    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1efabaab-7c52-45a8-88ab-aa701f749d89-kube-api-access-j52mp" (OuterVolumeSpecName: "kube-api-access-j52mp") pod "1efabaab-7c52-45a8-88ab-aa701f749d89" (UID: "1efabaab-7c52-45a8-88ab-aa701f749d89"). InnerVolumeSpecName "kube-api-access-j52mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:08:34.445762 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.445663    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bdf7e41-145a-4816-a198-3037a4552393-kube-api-access-fbm6z" (OuterVolumeSpecName: "kube-api-access-fbm6z") pod "5bdf7e41-145a-4816-a198-3037a4552393" (UID: "5bdf7e41-145a-4816-a198-3037a4552393"). InnerVolumeSpecName "kube-api-access-fbm6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:08:34.544173 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.544136    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbm6z\" (UniqueName: \"kubernetes.io/projected/5bdf7e41-145a-4816-a198-3037a4552393-kube-api-access-fbm6z\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:08:34.544173 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.544166    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1efabaab-7c52-45a8-88ab-aa701f749d89-plugins-config-volume\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:08:34.544173 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.544175    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j52mp\" (UniqueName: \"kubernetes.io/projected/1efabaab-7c52-45a8-88ab-aa701f749d89-kube-api-access-j52mp\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:08:34.544441 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.544183    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bdf7e41-145a-4816-a198-3037a4552393-plugins-config-volume\") on node \"ip-10-0-128-194.ec2.internal\" DevicePath \"\""
Jul 24 13:08:34.647122 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:34.647040    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="996e0a58-4ca8-41d7-b82e-fda2c909c539" path="/var/lib/kubelet/pods/996e0a58-4ca8-41d7-b82e-fda2c909c539/volumes"
Jul 24 13:08:35.331367 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:35.331331    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8"
Jul 24 13:08:35.331906 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:35.331366    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8" event={"ID":"5bdf7e41-145a-4816-a198-3037a4552393","Type":"ContainerDied","Data":"264deff23fefb69f301ebe46e33ca902db3603e89858d7cfb8d6f3a013fdca61"}
Jul 24 13:08:35.331906 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:35.331480    2581 scope.go:117] "RemoveContainer" containerID="c1b914b343b8bc144239fb67bab6044985a901ebd5eaadb50ee75dfba1dcb10d"
Jul 24 13:08:35.332919 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:35.332895    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt" event={"ID":"1efabaab-7c52-45a8-88ab-aa701f749d89","Type":"ContainerDied","Data":"37434ccce1e2466c4c7d71e6c8fd981eab3a4c7333aa2d5a26ba37ddbc50648d"}
Jul 24 13:08:35.333024 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:35.332925    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt"
Jul 24 13:08:35.340348 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:35.340331    2581 scope.go:117] "RemoveContainer" containerID="9d81f3f795316c75b92a0e1348df9128b6af661a427790939f5d6d14beb43062"
Jul 24 13:08:35.350678 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:35.350654    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8"]
Jul 24 13:08:35.353186 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:35.353163    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c6697a-5fbbd5f6-c6ns8"]
Jul 24 13:08:35.364274 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:35.364254    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt"]
Jul 24 13:08:35.366812 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:35.366790    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-c6697a-87cd49fc8-r9xnt"]
Jul 24 13:08:36.645579 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:36.645537    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1efabaab-7c52-45a8-88ab-aa701f749d89" path="/var/lib/kubelet/pods/1efabaab-7c52-45a8-88ab-aa701f749d89/volumes"
Jul 24 13:08:36.646076 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:36.645904    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bdf7e41-145a-4816-a198-3037a4552393" path="/var/lib/kubelet/pods/5bdf7e41-145a-4816-a198-3037a4552393/volumes"
Jul 24 13:08:37.003184 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:37.003096    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-w6p8w"]
Jul 24 13:08:38.504145 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:38.504107    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6697a-8bpg5"]
Jul 24 13:08:38.506896 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:38.506868    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c6697a-8bpg5"]
Jul 24 13:08:38.647594 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:38.647562    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cfc9441-8830-4c62-a4e2-923604f514f9" path="/var/lib/kubelet/pods/7cfc9441-8830-4c62-a4e2-923604f514f9/volumes"
Jul 24 13:08:52.316342 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:52.316266    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565c4d787c-v5dfd_eff6885f-c754-4a73-a93f-0b9ccd7de582/copy-manifests/0.log"
Jul 24 13:08:52.323856 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:52.323825    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565c4d787c-v5dfd_eff6885f-c754-4a73-a93f-0b9ccd7de582/manager/0.log"
Jul 24 13:08:53.056324 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:53.056294    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748300-5ccxz_4b57764e-81e2-4d59-ae3b-a7b3466b8f83/cleanup/0.log"
Jul 24 13:08:55.161194 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:55.161161    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vrkz8_30be1ce8-caf8-45b1-9be2-c50c3ce7f8ea/manager/0.log"
Jul 24 13:08:55.305674 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:55.305648    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-w6p8w_57515368-5175-44eb-a0f8-841f33ee7427/limitador/0.log"
Jul 24 13:08:56.100242 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:56.100207    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bfc95c995-v4dz5_477446d3-4478-4510-a2e7-59d1d88bd115/kube-auth-proxy/0.log"
Jul 24 13:08:57.730000 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:08:57.729970    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-rvhgs_76911653-0a8a-48b9-ae24-3d3b4fe3b355/main/0.log"
Jul 24 13:09:03.307107 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.307069    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bz6zh/must-gather-vlkpn"]
Jul 24 13:09:03.307532 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.307471    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1efabaab-7c52-45a8-88ab-aa701f749d89" containerName="payload-processing"
Jul 24 13:09:03.307532 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.307485    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1efabaab-7c52-45a8-88ab-aa701f749d89" containerName="payload-processing"
Jul 24 13:09:03.307532 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.307503    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bdf7e41-145a-4816-a198-3037a4552393" containerName="payload-pre-processing"
Jul 24 13:09:03.307532 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.307508    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bdf7e41-145a-4816-a198-3037a4552393" containerName="payload-pre-processing"
Jul 24 13:09:03.307532 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.307517    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cfc9441-8830-4c62-a4e2-923604f514f9" containerName="revoke-keys"
Jul 24 13:09:03.307532 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.307523    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cfc9441-8830-4c62-a4e2-923604f514f9" containerName="revoke-keys"
Jul 24 13:09:03.307734 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.307596    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cfc9441-8830-4c62-a4e2-923604f514f9" containerName="revoke-keys"
Jul 24 13:09:03.307734 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.307604    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bdf7e41-145a-4816-a198-3037a4552393" containerName="payload-pre-processing"
Jul 24 13:09:03.307734 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.307615    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1efabaab-7c52-45a8-88ab-aa701f749d89" containerName="payload-processing"
Jul 24 13:09:03.312305 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.312283    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bz6zh/must-gather-vlkpn"
Jul 24 13:09:03.314926 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.314904    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bz6zh\"/\"kube-root-ca.crt\""
Jul 24 13:09:03.315064 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.314905    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bz6zh\"/\"default-dockercfg-jsx6c\""
Jul 24 13:09:03.315276 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.315261    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bz6zh\"/\"openshift-service-ca.crt\""
Jul 24 13:09:03.319599 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.319575    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bz6zh/must-gather-vlkpn"]
Jul 24 13:09:03.416016 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.415969    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dqmn\" (UniqueName: \"kubernetes.io/projected/271028f7-8dbe-4b4b-b623-8689c520acf4-kube-api-access-9dqmn\") pod \"must-gather-vlkpn\" (UID: \"271028f7-8dbe-4b4b-b623-8689c520acf4\") " pod="openshift-must-gather-bz6zh/must-gather-vlkpn"
Jul 24 13:09:03.416203 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.416107    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/271028f7-8dbe-4b4b-b623-8689c520acf4-must-gather-output\") pod \"must-gather-vlkpn\" (UID: \"271028f7-8dbe-4b4b-b623-8689c520acf4\") " pod="openshift-must-gather-bz6zh/must-gather-vlkpn"
Jul 24 13:09:03.517587 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.517545    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dqmn\" (UniqueName: \"kubernetes.io/projected/271028f7-8dbe-4b4b-b623-8689c520acf4-kube-api-access-9dqmn\") pod \"must-gather-vlkpn\" (UID: \"271028f7-8dbe-4b4b-b623-8689c520acf4\") " pod="openshift-must-gather-bz6zh/must-gather-vlkpn"
Jul 24 13:09:03.517770 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.517641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/271028f7-8dbe-4b4b-b623-8689c520acf4-must-gather-output\") pod \"must-gather-vlkpn\" (UID: \"271028f7-8dbe-4b4b-b623-8689c520acf4\") " pod="openshift-must-gather-bz6zh/must-gather-vlkpn"
Jul 24 13:09:03.517988 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.517967    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/271028f7-8dbe-4b4b-b623-8689c520acf4-must-gather-output\") pod \"must-gather-vlkpn\" (UID: \"271028f7-8dbe-4b4b-b623-8689c520acf4\") " pod="openshift-must-gather-bz6zh/must-gather-vlkpn"
Jul 24 13:09:03.526000 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.525976    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dqmn\" (UniqueName: \"kubernetes.io/projected/271028f7-8dbe-4b4b-b623-8689c520acf4-kube-api-access-9dqmn\") pod \"must-gather-vlkpn\" (UID: \"271028f7-8dbe-4b4b-b623-8689c520acf4\") " pod="openshift-must-gather-bz6zh/must-gather-vlkpn"
Jul 24 13:09:03.622965 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.622880    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bz6zh/must-gather-vlkpn"
Jul 24 13:09:03.760234 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.760206    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bz6zh/must-gather-vlkpn"]
Jul 24 13:09:03.763111 ip-10-0-128-194 kubenswrapper[2581]: W0724 13:09:03.763082    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod271028f7_8dbe_4b4b_b623_8689c520acf4.slice/crio-c30d760be6d5709a8cc9b4ca9b7ef01a7e474a6930634ce26b82341641732ab6 WatchSource:0}: Error finding container c30d760be6d5709a8cc9b4ca9b7ef01a7e474a6930634ce26b82341641732ab6: Status 404 returned error can't find the container with id c30d760be6d5709a8cc9b4ca9b7ef01a7e474a6930634ce26b82341641732ab6
Jul 24 13:09:03.765068 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:03.765049    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:09:04.481747 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:04.481711    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bz6zh/must-gather-vlkpn" event={"ID":"271028f7-8dbe-4b4b-b623-8689c520acf4","Type":"ContainerStarted","Data":"c30d760be6d5709a8cc9b4ca9b7ef01a7e474a6930634ce26b82341641732ab6"}
Jul 24 13:09:05.488370 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:05.488330    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bz6zh/must-gather-vlkpn" event={"ID":"271028f7-8dbe-4b4b-b623-8689c520acf4","Type":"ContainerStarted","Data":"0550bc99a652363d0f2e176975ada7abc1c40b178548df874dbe42a7a69214f9"}
Jul 24 13:09:05.488841 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:05.488383    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bz6zh/must-gather-vlkpn" event={"ID":"271028f7-8dbe-4b4b-b623-8689c520acf4","Type":"ContainerStarted","Data":"3d7b9ca161927e41a1e98598e984ab0e9a8919c56ea57361d10f2c338529c3a3"}
Jul 24 13:09:05.509139 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:05.509077    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bz6zh/must-gather-vlkpn" podStartSLOduration=1.666478466 podStartE2EDuration="2.509054441s" totalImagesPullingTime="842.575975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:09:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:09:05.506657954 +0000 UTC m=+2977.461662719" watchObservedRunningTime="2026-07-24 13:09:05.509054441 +0000 UTC m=+2977.464059208"
Jul 24 13:09:06.293199 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:06.293170    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qrzvj_d98f89d0-11ef-4c6b-b973-64267283f1da/global-pull-secret-syncer/0.log"
Jul 24 13:09:06.465599 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:06.465560    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hrsjh_1dc4198c-786b-4dec-abb0-0838e3fab08a/konnectivity-agent/0.log"
Jul 24 13:09:06.491789 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:06.491760    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-194.ec2.internal_69ea8b15ea96888b6c85fb2084a0817d/haproxy/0.log"
Jul 24 13:09:11.813086 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:11.812720    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vrkz8_30be1ce8-caf8-45b1-9be2-c50c3ce7f8ea/manager/0.log"
Jul 24 13:09:11.838776 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:11.838559    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-w6p8w_57515368-5175-44eb-a0f8-841f33ee7427/limitador/0.log"
Jul 24 13:09:13.380820 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:13.380783    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54a7f8b4-055f-424b-87a5-d83333058107/alertmanager/0.log"
Jul 24 13:09:13.408298 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:13.408273    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54a7f8b4-055f-424b-87a5-d83333058107/config-reloader/0.log"
Jul 24 13:09:13.437824 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:13.437782    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54a7f8b4-055f-424b-87a5-d83333058107/kube-rbac-proxy-web/0.log"
Jul 24 13:09:13.463023 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:13.463000    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54a7f8b4-055f-424b-87a5-d83333058107/kube-rbac-proxy/0.log"
Jul 24 13:09:13.488557 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:13.488471    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54a7f8b4-055f-424b-87a5-d83333058107/kube-rbac-proxy-metric/0.log"
Jul 24 13:09:13.511719 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:13.511683    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54a7f8b4-055f-424b-87a5-d83333058107/prom-label-proxy/0.log"
Jul 24 13:09:13.537200 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:13.537176    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54a7f8b4-055f-424b-87a5-d83333058107/init-config-reloader/0.log"
Jul 24 13:09:13.624211 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:13.624158    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-cbkfr_18cdccc7-5d25-4e5b-af0e-b0a40d851053/kube-state-metrics/0.log"
Jul 24 13:09:13.651934 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:13.651908    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-cbkfr_18cdccc7-5d25-4e5b-af0e-b0a40d851053/kube-rbac-proxy-main/0.log"
Jul 24 13:09:13.677449 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:13.677406    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-cbkfr_18cdccc7-5d25-4e5b-af0e-b0a40d851053/kube-rbac-proxy-self/0.log"
Jul 24 13:09:13.711198 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:13.711156    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-86866dff5f-8ph8r_a1c15d37-5a52-4f35-8f0a-fa629c497ad7/metrics-server/0.log"
Jul 24 13:09:13.761088 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:13.760780    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-jmsvs_3ab6ce6c-61a7-49a6-8e86-fc76f9c2e041/monitoring-plugin/0.log"
Jul 24 13:09:13.903911 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:13.903883    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rrtq2_99c008bb-acfe-480a-a07c-9bccdce3d34b/node-exporter/0.log"
Jul 24 13:09:13.928940 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:13.928915    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rrtq2_99c008bb-acfe-480a-a07c-9bccdce3d34b/kube-rbac-proxy/0.log"
Jul 24 13:09:13.957438 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:13.957106    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rrtq2_99c008bb-acfe-480a-a07c-9bccdce3d34b/init-textfile/0.log"
Jul 24 13:09:14.070041 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.069999    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-q22x5_215ec8d5-e824-4a7e-9b89-4b9ee4127682/kube-rbac-proxy-main/0.log"
Jul 24 13:09:14.093795 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.093762    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-q22x5_215ec8d5-e824-4a7e-9b89-4b9ee4127682/kube-rbac-proxy-self/0.log"
Jul 24 13:09:14.117827 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.117794    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-q22x5_215ec8d5-e824-4a7e-9b89-4b9ee4127682/openshift-state-metrics/0.log"
Jul 24 13:09:14.165005 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.164961    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_997cf464-b4ac-4dca-a03b-2c5026a6c2b2/prometheus/0.log"
Jul 24 13:09:14.198448 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.198406    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_997cf464-b4ac-4dca-a03b-2c5026a6c2b2/config-reloader/0.log"
Jul 24 13:09:14.224022 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.223993    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_997cf464-b4ac-4dca-a03b-2c5026a6c2b2/thanos-sidecar/0.log"
Jul 24 13:09:14.252753 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.252725    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_997cf464-b4ac-4dca-a03b-2c5026a6c2b2/kube-rbac-proxy-web/0.log"
Jul 24 13:09:14.278971 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.278888    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_997cf464-b4ac-4dca-a03b-2c5026a6c2b2/kube-rbac-proxy/0.log"
Jul 24 13:09:14.304347 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.304312    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_997cf464-b4ac-4dca-a03b-2c5026a6c2b2/kube-rbac-proxy-thanos/0.log"
Jul 24 13:09:14.327803 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.327719    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_997cf464-b4ac-4dca-a03b-2c5026a6c2b2/init-config-reloader/0.log"
Jul 24 13:09:14.358760 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.358726    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-2ckvk_91263e28-33ba-4bcf-854d-f3a614be63e2/prometheus-operator/0.log"
Jul 24 13:09:14.382004 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.381970    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-2ckvk_91263e28-33ba-4bcf-854d-f3a614be63e2/kube-rbac-proxy/0.log"
Jul 24 13:09:14.478662 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.478628    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77c9c7c646-cznc8_9f736943-1b3c-493f-a2d0-f767b1794d2a/telemeter-client/0.log"
Jul 24 13:09:14.502204 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.502156    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77c9c7c646-cznc8_9f736943-1b3c-493f-a2d0-f767b1794d2a/reload/0.log"
Jul 24 13:09:14.526332 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.526296    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77c9c7c646-cznc8_9f736943-1b3c-493f-a2d0-f767b1794d2a/kube-rbac-proxy/0.log"
Jul 24 13:09:14.559939 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.559910    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85c5b8bcfb-bkqjk_789a6a57-4d93-47d3-abd7-201f0c2d53eb/thanos-query/0.log"
Jul 24 13:09:14.590130 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.590041    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85c5b8bcfb-bkqjk_789a6a57-4d93-47d3-abd7-201f0c2d53eb/kube-rbac-proxy-web/0.log"
Jul 24 13:09:14.619995 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.619962    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85c5b8bcfb-bkqjk_789a6a57-4d93-47d3-abd7-201f0c2d53eb/kube-rbac-proxy/0.log"
Jul 24 13:09:14.642869 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.642840    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85c5b8bcfb-bkqjk_789a6a57-4d93-47d3-abd7-201f0c2d53eb/prom-label-proxy/0.log"
Jul 24 13:09:14.667784 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.667760    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85c5b8bcfb-bkqjk_789a6a57-4d93-47d3-abd7-201f0c2d53eb/kube-rbac-proxy-rules/0.log"
Jul 24 13:09:14.693702 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:14.693676    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85c5b8bcfb-bkqjk_789a6a57-4d93-47d3-abd7-201f0c2d53eb/kube-rbac-proxy-metrics/0.log"
Jul 24 13:09:16.715693 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:16.715654    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"]
Jul 24 13:09:16.721660 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:16.721628    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"
Jul 24 13:09:16.734647 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:16.734609    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"]
Jul 24 13:09:16.779086 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:16.779041    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b83d84cb-c093-4d65-9777-db68ba737ce0-proc\") pod \"perf-node-gather-daemonset-4zkpm\" (UID: \"b83d84cb-c093-4d65-9777-db68ba737ce0\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"
Jul 24 13:09:16.779300 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:16.779118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz4kj\" (UniqueName: \"kubernetes.io/projected/b83d84cb-c093-4d65-9777-db68ba737ce0-kube-api-access-hz4kj\") pod \"perf-node-gather-daemonset-4zkpm\" (UID: \"b83d84cb-c093-4d65-9777-db68ba737ce0\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"
Jul 24 13:09:16.779300 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:16.779182    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b83d84cb-c093-4d65-9777-db68ba737ce0-podres\") pod \"perf-node-gather-daemonset-4zkpm\" (UID: \"b83d84cb-c093-4d65-9777-db68ba737ce0\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"
Jul 24 13:09:16.779300 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:16.779209    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b83d84cb-c093-4d65-9777-db68ba737ce0-lib-modules\") pod \"perf-node-gather-daemonset-4zkpm\" (UID: \"b83d84cb-c093-4d65-9777-db68ba737ce0\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"
Jul 24 13:09:16.779300 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:16.779236    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b83d84cb-c093-4d65-9777-db68ba737ce0-sys\") pod \"perf-node-gather-daemonset-4zkpm\" (UID: \"b83d84cb-c093-4d65-9777-db68ba737ce0\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"
Jul 24 13:09:16.880223 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:16.880189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b83d84cb-c093-4d65-9777-db68ba737ce0-proc\") pod \"perf-node-gather-daemonset-4zkpm\" (UID: \"b83d84cb-c093-4d65-9777-db68ba737ce0\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"
Jul 24 13:09:16.880404 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:16.880254    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz4kj\" (UniqueName: \"kubernetes.io/projected/b83d84cb-c093-4d65-9777-db68ba737ce0-kube-api-access-hz4kj\") pod \"perf-node-gather-daemonset-4zkpm\" (UID: \"b83d84cb-c093-4d65-9777-db68ba737ce0\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"
Jul 24 13:09:16.880404 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:16.880320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b83d84cb-c093-4d65-9777-db68ba737ce0-podres\") pod \"perf-node-gather-daemonset-4zkpm\" (UID: \"b83d84cb-c093-4d65-9777-db68ba737ce0\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"
Jul 24 13:09:16.880404 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:16.880336    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b83d84cb-c093-4d65-9777-db68ba737ce0-proc\") pod \"perf-node-gather-daemonset-4zkpm\" (UID: \"b83d84cb-c093-4d65-9777-db68ba737ce0\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"
Jul 24 13:09:16.880404 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:16.880344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b83d84cb-c093-4d65-9777-db68ba737ce0-lib-modules\") pod \"perf-node-gather-daemonset-4zkpm\" (UID: \"b83d84cb-c093-4d65-9777-db68ba737ce0\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"
Jul 24 13:09:16.880404 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:16.880382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b83d84cb-c093-4d65-9777-db68ba737ce0-sys\") pod \"perf-node-gather-daemonset-4zkpm\" (UID: \"b83d84cb-c093-4d65-9777-db68ba737ce0\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"
Jul 24 13:09:16.880702 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:16.880493    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b83d84cb-c093-4d65-9777-db68ba737ce0-lib-modules\") pod \"perf-node-gather-daemonset-4zkpm\" (UID: \"b83d84cb-c093-4d65-9777-db68ba737ce0\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"
Jul 24 13:09:16.880702 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:16.880525    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b83d84cb-c093-4d65-9777-db68ba737ce0-sys\") pod \"perf-node-gather-daemonset-4zkpm\" (UID: \"b83d84cb-c093-4d65-9777-db68ba737ce0\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"
Jul 24 13:09:16.880702 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:16.880523    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b83d84cb-c093-4d65-9777-db68ba737ce0-podres\") pod \"perf-node-gather-daemonset-4zkpm\" (UID: \"b83d84cb-c093-4d65-9777-db68ba737ce0\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"
Jul 24 13:09:16.896644 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:16.896609    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz4kj\" (UniqueName: \"kubernetes.io/projected/b83d84cb-c093-4d65-9777-db68ba737ce0-kube-api-access-hz4kj\") pod \"perf-node-gather-daemonset-4zkpm\" (UID: \"b83d84cb-c093-4d65-9777-db68ba737ce0\") " pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"
Jul 24 13:09:17.040772 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:17.040737    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"
Jul 24 13:09:17.230676 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:17.229849    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"]
Jul 24 13:09:17.233199 ip-10-0-128-194 kubenswrapper[2581]: W0724 13:09:17.233152    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb83d84cb_c093_4d65_9777_db68ba737ce0.slice/crio-f77a7c04aac21223d78f1c61cdc26c856dfd942c3b757ac570767a625b0f5c6a WatchSource:0}: Error finding container f77a7c04aac21223d78f1c61cdc26c856dfd942c3b757ac570767a625b0f5c6a: Status 404 returned error can't find the container with id f77a7c04aac21223d78f1c61cdc26c856dfd942c3b757ac570767a625b0f5c6a
Jul 24 13:09:17.584555 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:17.584463    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm" event={"ID":"b83d84cb-c093-4d65-9777-db68ba737ce0","Type":"ContainerStarted","Data":"795f62fd4045a2142ed5b497639685511090a987ea64cf69b3f0b5c3060be15c"}
Jul 24 13:09:17.584555 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:17.584533    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm" event={"ID":"b83d84cb-c093-4d65-9777-db68ba737ce0","Type":"ContainerStarted","Data":"f77a7c04aac21223d78f1c61cdc26c856dfd942c3b757ac570767a625b0f5c6a"}
Jul 24 13:09:17.585366 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:17.585337    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"
Jul 24 13:09:17.603923 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:17.603861    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm" podStartSLOduration=1.6038404499999999 podStartE2EDuration="1.60384045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:09:17.601115675 +0000 UTC m=+2989.556120438" watchObservedRunningTime="2026-07-24 13:09:17.60384045 +0000 UTC m=+2989.558845214"
Jul 24 13:09:19.044994 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:19.044963    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8dqxn_ef8ade5f-a030-41c8-b643-6f9ef781a148/dns/0.log"
Jul 24 13:09:19.068552 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:19.068517    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8dqxn_ef8ade5f-a030-41c8-b643-6f9ef781a148/kube-rbac-proxy/0.log"
Jul 24 13:09:19.192848 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:19.192821    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jmmcs_0385bad7-43a0-4ee3-b116-0e89d51f30f7/dns-node-resolver/0.log"
Jul 24 13:09:19.799876 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:19.799845    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sj8p9_c74cea6b-83f7-4615-918a-d27808ed55ee/node-ca/0.log"
Jul 24 13:09:21.042050 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:21.042023    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bfc95c995-v4dz5_477446d3-4478-4510-a2e7-59d1d88bd115/kube-auth-proxy/0.log"
Jul 24 13:09:22.024738 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:22.024700    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-564ct_dbc2bb0f-05f3-4758-bcc8-71245bf72991/serve-healthcheck-canary/0.log"
Jul 24 13:09:22.667935 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:22.667908    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rzcwk_aa4a90db-e263-4e0d-ad83-9b0a60b282a0/kube-rbac-proxy/0.log"
Jul 24 13:09:22.689951 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:22.689926    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rzcwk_aa4a90db-e263-4e0d-ad83-9b0a60b282a0/exporter/0.log"
Jul 24 13:09:22.713099 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:22.713068    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rzcwk_aa4a90db-e263-4e0d-ad83-9b0a60b282a0/extractor/0.log"
Jul 24 13:09:24.618174 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:24.618135    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bz6zh/perf-node-gather-daemonset-4zkpm"
Jul 24 13:09:25.851501 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:25.851474    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565c4d787c-v5dfd_eff6885f-c754-4a73-a93f-0b9ccd7de582/manager/0.log"
Jul 24 13:09:25.874323 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:25.874297    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565c4d787c-v5dfd_eff6885f-c754-4a73-a93f-0b9ccd7de582/copy-manifests/0.log"
Jul 24 13:09:27.256835 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:27.256750    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-d4b69f9c5-9h6f6_2c46d08c-90dd-4a01-8a5f-fd8985444c68/manager/0.log"
Jul 24 13:09:27.305849 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:27.305787    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-ck7q4_3a282e7d-5bab-4d0d-b155-80e2d435804d/openshift-lws-operator/0.log"
Jul 24 13:09:27.668525 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:27.668496    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bfc95c995-v4dz5_477446d3-4478-4510-a2e7-59d1d88bd115/kube-auth-proxy/0.log"
Jul 24 13:09:29.021701 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:29.021627    2581 scope.go:117] "RemoveContainer" containerID="03acc44d416a2159015bc3effff14a3e0c8d3bda935ed27d898adc687cf7a8d0"
Jul 24 13:09:29.536052 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:29.536026    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bfc95c995-v4dz5_477446d3-4478-4510-a2e7-59d1d88bd115/kube-auth-proxy/0.log"
Jul 24 13:09:33.922735 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:33.922698    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bk4w6_f42ba11d-0506-4915-bc77-d52d5fd3ea4a/kube-multus-additional-cni-plugins/0.log"
Jul 24 13:09:33.947902 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:33.947864    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bk4w6_f42ba11d-0506-4915-bc77-d52d5fd3ea4a/egress-router-binary-copy/0.log"
Jul 24 13:09:33.972289 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:33.972258    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bk4w6_f42ba11d-0506-4915-bc77-d52d5fd3ea4a/cni-plugins/0.log"
Jul 24 13:09:33.995141 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:33.995116    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bk4w6_f42ba11d-0506-4915-bc77-d52d5fd3ea4a/bond-cni-plugin/0.log"
Jul 24 13:09:34.017673 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:34.017650    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bk4w6_f42ba11d-0506-4915-bc77-d52d5fd3ea4a/routeoverride-cni/0.log"
Jul 24 13:09:34.040926 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:34.040895    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bk4w6_f42ba11d-0506-4915-bc77-d52d5fd3ea4a/whereabouts-cni-bincopy/0.log"
Jul 24 13:09:34.067151 ip-10-0-128-194 kubenswrapper[2581]: I0724 13:09:34.067123    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bk4w6_f42ba11d-0506-4915-bc77-d52d5fd3ea4a/whereabouts-cni/0.log"