Jul 28 20:45:40.759422 ip-10-0-134-170 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 20:45:40.759435 ip-10-0-134-170 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 20:45:40.759445 ip-10-0-134-170 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 20:45:40.759766 ip-10-0-134-170 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 20:45:51.008967 ip-10-0-134-170 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 20:45:51.008988 ip-10-0-134-170 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot dea4d9557c18493f83642995a34ef3a8 --
Jul 28 20:47:24.647941 ip-10-0-134-170 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 20:47:25.096778 ip-10-0-134-170 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 20:47:25.096778 ip-10-0-134-170 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 20:47:25.097584 ip-10-0-134-170 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 20:47:25.097584 ip-10-0-134-170 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 20:47:25.097584 ip-10-0-134-170 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 20:47:25.098175 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.097984    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 20:47:25.103131 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103114    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:47:25.103131 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103131    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:47:25.103210 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103136    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:47:25.103210 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103139    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:47:25.103210 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103143    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:47:25.103210 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103146    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:47:25.103210 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103148    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:47:25.103210 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103151    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:47:25.103210 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103155    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:47:25.103210 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103159    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:47:25.103210 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103162    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:47:25.103210 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103164    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:47:25.103210 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103167    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:47:25.103210 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103170    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:47:25.103210 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103172    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:47:25.103210 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103175    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:47:25.103210 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103178    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:47:25.103210 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103181    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:47:25.103210 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103183    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:47:25.103210 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103186    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:47:25.103210 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103189    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:47:25.103681 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103191    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:47:25.103681 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103194    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:47:25.103681 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103200    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:47:25.103681 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103202    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:47:25.103681 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103205    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:47:25.103681 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103208    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:47:25.103681 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103213    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:47:25.103681 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103217    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:47:25.103681 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103220    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:47:25.103681 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103224    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:47:25.103681 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103226    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:47:25.103681 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103230    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:47:25.103681 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103233    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:47:25.103681 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103236    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:47:25.103681 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103238    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:47:25.103681 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103241    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:47:25.103681 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103243    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:47:25.103681 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103246    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:47:25.103681 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103248    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:47:25.104165 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103251    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:47:25.104165 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103254    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:47:25.104165 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103256    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:47:25.104165 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103259    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:47:25.104165 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103261    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:47:25.104165 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103264    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:47:25.104165 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103267    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:47:25.104165 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103272    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:47:25.104165 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103275    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:47:25.104165 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103277    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:47:25.104165 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103280    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:47:25.104165 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103283    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:47:25.104165 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103285    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:47:25.104165 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103289    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:47:25.104165 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103292    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:47:25.104165 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103295    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:47:25.104165 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103297    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:47:25.104165 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103300    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:47:25.104165 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103302    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:47:25.104165 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103305    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:47:25.104676 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103307    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:47:25.104676 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103311    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:47:25.104676 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103313    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:47:25.104676 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103316    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:47:25.104676 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103319    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:47:25.104676 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103321    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:47:25.104676 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103324    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:47:25.104676 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103328    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:47:25.104676 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103330    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:47:25.104676 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103333    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:47:25.104676 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103335    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:47:25.104676 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103337    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:47:25.104676 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103340    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:47:25.104676 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103342    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:47:25.104676 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103345    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:47:25.104676 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103347    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:47:25.104676 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103350    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:47:25.104676 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103353    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:47:25.104676 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103355    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:47:25.104676 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103357    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:47:25.105167 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103361    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:47:25.105167 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103364    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:47:25.105167 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103366    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:47:25.105167 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103369    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:47:25.105167 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103373    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:47:25.105167 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103378    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:47:25.105167 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103381    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:47:25.105167 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103384    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:47:25.105167 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103386    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:47:25.105167 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103389    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:47:25.105167 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103392    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:47:25.105167 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103394    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:47:25.105167 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103397    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:47:25.105167 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103400    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:47:25.105167 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103402    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:47:25.105167 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103405    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:47:25.105167 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103409    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:47:25.105167 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103412    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:47:25.105167 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.103415    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:47:25.105802 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104135    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:47:25.105802 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104143    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:47:25.105802 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104147    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:47:25.105802 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104150    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:47:25.105802 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104153    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:47:25.105802 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104155    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:47:25.105802 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104158    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:47:25.105802 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104160    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:47:25.105802 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104163    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:47:25.105802 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104166    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:47:25.105802 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104169    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:47:25.105802 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104172    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:47:25.105802 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104175    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:47:25.105802 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104178    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:47:25.105802 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104182    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:47:25.105802 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104185    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:47:25.105802 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104187    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:47:25.105802 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104191    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:47:25.105802 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104194    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:47:25.105802 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104197    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:47:25.106421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104199    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:47:25.106421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104202    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:47:25.106421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104204    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:47:25.106421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104207    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:47:25.106421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104210    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:47:25.106421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104212    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:47:25.106421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104215    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:47:25.106421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104217    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:47:25.106421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104221    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:47:25.106421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104223    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:47:25.106421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104226    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:47:25.106421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104228    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:47:25.106421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104231    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:47:25.106421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104234    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:47:25.106421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104236    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:47:25.106421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104238    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:47:25.106421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104241    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:47:25.106421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104243    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:47:25.106421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104246    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:47:25.106421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104248    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:47:25.106936 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104251    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:47:25.106936 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104253    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:47:25.106936 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104256    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:47:25.106936 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104260    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:47:25.106936 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104262    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:47:25.106936 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104265    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:47:25.106936 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104267    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:47:25.106936 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104270    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:47:25.106936 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104273    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:47:25.106936 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104277    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:47:25.106936 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104279    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:47:25.106936 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104282    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:47:25.106936 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104285    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:47:25.106936 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104287    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:47:25.106936 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104290    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:47:25.106936 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104293    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:47:25.106936 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104297    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:47:25.106936 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104300    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:47:25.107386 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104302    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:47:25.107386 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104305    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:47:25.107386 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104310    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:47:25.107386 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104313    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:47:25.107386 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104316    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:47:25.107386 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104319    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:47:25.107386 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104322    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:47:25.107386 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104325    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:47:25.107386 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104328    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:47:25.107386 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104330    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:47:25.107386 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104333    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:47:25.107386 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104336    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:47:25.107386 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104338    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:47:25.107386 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104341    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:47:25.107386 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104343    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:47:25.107386 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104345    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:47:25.107386 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104348    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:47:25.107386 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104351    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:47:25.107386 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104355    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104358    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104361    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104364    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104367    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104369    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104372    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104375    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104377    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104380    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104382    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104385    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104387    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104389    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104392    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104394    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104397    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104400    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104402    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104404    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104408    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:47:25.107882 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104410    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:47:25.108401 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104412    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:47:25.108401 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104851    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:47:25.108401 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104857    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:47:25.108401 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104860    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:47:25.108401 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104863    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:47:25.108401 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104866    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:47:25.108401 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104869    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:47:25.108401 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104871    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:47:25.108401 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104876    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:47:25.108401 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104879    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:47:25.108401 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104882    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:47:25.108401 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104885    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:47:25.108401 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104888    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:47:25.108401 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104891    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:47:25.108401 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104903    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:47:25.108401 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104906    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:47:25.108401 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104909    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:47:25.108401 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104911    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:47:25.108401 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104914    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:47:25.108871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104917    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:47:25.108871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104919    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:47:25.108871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104922    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:47:25.108871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104924    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:47:25.108871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104927    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:47:25.108871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104930    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:47:25.108871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104932    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:47:25.108871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104935    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:47:25.108871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104937    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:47:25.108871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104940    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:47:25.108871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104942    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:47:25.108871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104945    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:47:25.108871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104948    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:47:25.108871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104952    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:47:25.108871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104955    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:47:25.108871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104958    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:47:25.108871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104960    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:47:25.108871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104963    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:47:25.108871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104966    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:47:25.108871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104968    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:47:25.109359 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104971    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:47:25.109359 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104975    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:47:25.109359 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104978    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:47:25.109359 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104981    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:47:25.109359 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104984    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:47:25.109359 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104986    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:47:25.109359 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104989    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:47:25.109359 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104992    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:47:25.109359 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.104998    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:47:25.109359 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105001    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:47:25.109359 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105003    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:47:25.109359 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105006    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:47:25.109359 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105009    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:47:25.109359 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105012    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:47:25.109359 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105014    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:47:25.109359 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105017    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:47:25.109359 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105019    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:47:25.109359 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105022    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:47:25.109359 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105025    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:47:25.109359 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105027    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:47:25.109858 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105030    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:47:25.109858 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105033    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:47:25.109858 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105037    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:47:25.109858 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105040    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:47:25.109858 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105042    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:47:25.109858 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105045    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:47:25.109858 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105047    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:47:25.109858 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105050    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:47:25.109858 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105052    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:47:25.109858 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105054    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:47:25.109858 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105057    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:47:25.109858 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105059    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:47:25.109858 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105062    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:47:25.109858 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105068    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:47:25.109858 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105072    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:47:25.109858 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105075    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:47:25.109858 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105077    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:47:25.109858 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105080    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:47:25.109858 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105082    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:47:25.110340 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105085    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:47:25.110340 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105088    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:47:25.110340 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105091    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:47:25.110340 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105094    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:47:25.110340 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105096    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:47:25.110340 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105099    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:47:25.110340 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105101    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:47:25.110340 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105104    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:47:25.110340 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105106    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:47:25.110340 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105109    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:47:25.110340 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105111    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:47:25.110340 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105114    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:47:25.110340 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105117    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:47:25.110340 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105120    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:47:25.110340 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105124    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:47:25.110340 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105126    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:47:25.110340 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105129    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:47:25.110340 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105132    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:47:25.110340 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105135    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105137    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105140    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105143    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105220    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105227    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105234    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105238    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105245    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105249    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105253    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105257    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105261    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105264    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105267    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105270    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105273    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105278    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105281    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105284    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105287    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105290    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105294    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105297    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 20:47:25.110856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105300    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105303    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105306    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105314    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105317    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105320    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105324    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105326    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105329    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105332    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105335    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105338    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105342    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105345    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105348    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105351    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105355    2565 flags.go:64] FLAG: --enable-server="true"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105359    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105364    2565 flags.go:64] FLAG: --event-burst="100"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105367    2565 flags.go:64] FLAG: --event-qps="50"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105370    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105373    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105375    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105379    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 20:47:25.111460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105382    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105385    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105389    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105392    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105395    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105398    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105401    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105404    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105406    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105410    2565 flags.go:64] FLAG: --feature-gates=""
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105413    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105417    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105420    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105423    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105426    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105429    2565 flags.go:64] FLAG: --help="false"
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105432    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-134-170.ec2.internal"
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105435    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105437    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105440    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105444    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105447    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105450    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 20:47:25.112062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105452    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105459    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105463    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105466    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105469    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105471    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105474    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105477    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105480    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105483    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105486    2565 flags.go:64] FLAG: --lock-file=""
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105488    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105492    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105495    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105501    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105504    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105506    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105509    2565 flags.go:64] FLAG: --logging-format="text"
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105512    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105515    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105518    2565 flags.go:64] FLAG: --manifest-url=""
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105521    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105525    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105528    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105532    2565 flags.go:64] FLAG: --max-pods="110"
Jul 28 20:47:25.112663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105535    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105538    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105541    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105544    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105547    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105550    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105553    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105559    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105565    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105568    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105571    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105574    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105598    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105602    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105606    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105609    2565 flags.go:64] FLAG: --port="10250"
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105611    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105614    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fbebf2e46c7f4288"
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105617    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105620    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105624    2565 flags.go:64] FLAG: --register-node="true"
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105627    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105644    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105648    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 28 20:47:25.113356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105651    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105653    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105657    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105660    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105663    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105666    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105669    2565 flags.go:64] FLAG: --runonce="false"
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105671    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105674    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105677    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105680    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105683    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105686    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105689    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105692    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105695    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105702    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105705    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105708    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105711    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105714    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105716    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105722    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105725    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105728    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 20:47:25.113983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105732    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105735    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105738    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105741    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105750    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105753    2565 flags.go:64] FLAG: --v="2"
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105757    2565 flags.go:64] FLAG: --version="false"
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105762    2565 flags.go:64] FLAG: --vmodule=""
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105765    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.105768    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105893    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105897    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105900    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105904    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105908    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105911    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105914    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105917    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105920    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105922    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105925    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:47:25.114650 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105927    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:47:25.115184 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105930    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:47:25.115184 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105935    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:47:25.115184 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105938    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:47:25.115184 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105940    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:47:25.115184 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105943    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:47:25.115184 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105945    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:47:25.115184 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105948    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:47:25.115184 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105951    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:47:25.115184 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105953    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:47:25.115184 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105956    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:47:25.115184 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105958    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:47:25.115184 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105961    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:47:25.115184 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105963    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:47:25.115184 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105966    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:47:25.115184 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105968    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:47:25.115184 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105972    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:47:25.115184 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105974    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:47:25.115184 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105977    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:47:25.115184 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105979    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:47:25.115184 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105982    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:47:25.115698 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105984    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:47:25.115698 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105987    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:47:25.115698 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105989    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:47:25.115698 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105992    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:47:25.115698 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105994    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:47:25.115698 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105997    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:47:25.115698 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.105999    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:47:25.115698 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106002    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:47:25.115698 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106004    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:47:25.115698 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106007    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:47:25.115698 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106009    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:47:25.115698 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106012    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:47:25.115698 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106014    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:47:25.115698 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106020    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:47:25.115698 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106022    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:47:25.115698 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106025    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:47:25.115698 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106029    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:47:25.115698 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106032    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:47:25.115698 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106035    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:47:25.116182 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106038    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:47:25.116182 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106040    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:47:25.116182 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106043    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:47:25.116182 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106046    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:47:25.116182 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106048    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:47:25.116182 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106051    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:47:25.116182 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106054    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:47:25.116182 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106056    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:47:25.116182 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106059    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:47:25.116182 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106062    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:47:25.116182 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106064    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:47:25.116182 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106067    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:47:25.116182 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106070    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:47:25.116182 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106072    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:47:25.116182 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106075    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:47:25.116182 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106078    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:47:25.116182 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106080    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:47:25.116182 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106083    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:47:25.116182 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106085    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:47:25.116182 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106089    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:47:25.116696 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106092    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:47:25.116696 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106095    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:47:25.116696 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106098    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:47:25.116696 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106100    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:47:25.116696 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106103    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:47:25.116696 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106106    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:47:25.116696 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106111    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:47:25.116696 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106114    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:47:25.116696 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106116    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:47:25.116696 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106119    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:47:25.116696 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106128    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:47:25.116696 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106132    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:47:25.116696 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106135    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:47:25.116696 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106137    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:47:25.116696 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106140    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:47:25.116696 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106143    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:47:25.116696 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106145    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:47:25.116696 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106148    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:47:25.116696 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106151    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:47:25.116696 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106153    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:47:25.117201 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106156    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:47:25.117201 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106159    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:47:25.117201 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106162    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:47:25.117201 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106165    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:47:25.117201 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106167    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:47:25.117201 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106170    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:47:25.117201 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106172    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:47:25.117201 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.106175    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:47:25.117201 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.106952    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 20:47:25.117201 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.114271    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 20:47:25.117201 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.114286    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 20:47:25.117201 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.115141    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 20:47:25.117201 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.115149    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 20:47:25.117201 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115209    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:47:25.117201 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115215    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:47:25.117599 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115218    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:47:25.117599 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115222    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:47:25.117599 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115224    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:47:25.117599 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115228    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:47:25.117599 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115230    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:47:25.117599 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115233    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:47:25.117599 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115236    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:47:25.117599 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115239    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:47:25.117599 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115241    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:47:25.117599 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115244    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:47:25.117599 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115246    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:47:25.117599 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115249    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:47:25.117599 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115252    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:47:25.117599 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115255    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:47:25.117599 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115257    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:47:25.117599 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115260    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:47:25.117599 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115264    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:47:25.117599 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115268    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:47:25.117599 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115271    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:47:25.117599 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115275    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:47:25.118108 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115278    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:47:25.118108 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115280    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:47:25.118108 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115283    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:47:25.118108 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115285    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:47:25.118108 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115288    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:47:25.118108 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115292    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:47:25.118108 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115295    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:47:25.118108 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115300    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:47:25.118108 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115303    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:47:25.118108 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115306    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:47:25.118108 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115309    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:47:25.118108 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115312    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:47:25.118108 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115314    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:47:25.118108 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115317    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:47:25.118108 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115320    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:47:25.118108 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115322    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:47:25.118108 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115325    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:47:25.118108 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115327    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:47:25.118108 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115331    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:47:25.118587 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115335    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:47:25.118587 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115338    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:47:25.118587 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115340    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:47:25.118587 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115343    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:47:25.118587 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115345    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:47:25.118587 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115349    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:47:25.118587 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115352    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:47:25.118587 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115354    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:47:25.118587 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115357    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:47:25.118587 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115360    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:47:25.118587 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115362    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:47:25.118587 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115365    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:47:25.118587 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115368    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:47:25.118587 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115371    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:47:25.118587 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115374    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:47:25.118587 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115377    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:47:25.118587 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115379    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:47:25.118587 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115382    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:47:25.118587 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115384    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:47:25.119079 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115387    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:47:25.119079 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115389    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:47:25.119079 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115392    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:47:25.119079 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115394    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:47:25.119079 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115397    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:47:25.119079 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115399    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:47:25.119079 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115402    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:47:25.119079 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115404    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:47:25.119079 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115407    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:47:25.119079 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115409    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:47:25.119079 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115412    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:47:25.119079 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115414    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:47:25.119079 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115417    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:47:25.119079 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115419    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:47:25.119079 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115422    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:47:25.119079 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115424    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:47:25.119079 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115427    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:47:25.119079 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115430    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:47:25.119079 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115433    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:47:25.119079 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115436    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:47:25.119560 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115439    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:47:25.119560 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115441    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:47:25.119560 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115444    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:47:25.119560 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115446    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:47:25.119560 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115449    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:47:25.119560 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115452    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:47:25.119560 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115455    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:47:25.119560 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115458    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:47:25.119560 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115460    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:47:25.119560 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115463    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:47:25.119560 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115466    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:47:25.119560 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115468    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:47:25.119560 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115471    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:47:25.119560 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115473    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:47:25.119560 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115476    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:47:25.119560 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115478    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:47:25.119560 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115480    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:47:25.119560 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115483    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:47:25.119560 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115486    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:47:25.120073 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.115490    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 20:47:25.120073 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115604    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 20:47:25.120073 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115609    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 20:47:25.120073 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115612    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 20:47:25.120073 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115615    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 20:47:25.120073 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115618    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 20:47:25.120073 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115621    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 20:47:25.120073 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115623    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 20:47:25.120073 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115628    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 20:47:25.120073 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115644    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 20:47:25.120073 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115649    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 20:47:25.120073 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115653    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 20:47:25.120073 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115656    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 20:47:25.120073 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115659    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 20:47:25.120421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115661    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 20:47:25.120421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115664    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 20:47:25.120421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115667    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 20:47:25.120421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115669    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 20:47:25.120421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115672    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 20:47:25.120421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115675    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 20:47:25.120421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115678    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 20:47:25.120421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115682    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 20:47:25.120421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115685    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 20:47:25.120421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115688    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 20:47:25.120421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115691    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 20:47:25.120421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115693    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 20:47:25.120421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115696    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 20:47:25.120421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115700    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 20:47:25.120421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115703    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 20:47:25.120421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115706    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 20:47:25.120421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115709    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 20:47:25.120421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115712    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 20:47:25.120421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115715    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 20:47:25.120421 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115717    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 20:47:25.120962 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115721    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 20:47:25.120962 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115724    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 20:47:25.120962 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115727    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 20:47:25.120962 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115730    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 20:47:25.120962 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115733    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 20:47:25.120962 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115735    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 20:47:25.120962 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115738    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 20:47:25.120962 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115740    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 20:47:25.120962 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115743    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 20:47:25.120962 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115746    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 20:47:25.120962 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115748    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 20:47:25.120962 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115751    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 20:47:25.120962 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115753    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 20:47:25.120962 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115756    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 20:47:25.120962 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115758    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 20:47:25.120962 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115761    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 20:47:25.120962 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115764    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 20:47:25.120962 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115767    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 20:47:25.120962 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115771    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 20:47:25.120962 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115773    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 20:47:25.121446 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115780    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 20:47:25.121446 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115782    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 20:47:25.121446 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115785    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 20:47:25.121446 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115788    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 20:47:25.121446 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115790    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 20:47:25.121446 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115793    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 20:47:25.121446 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115795    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 20:47:25.121446 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115798    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 20:47:25.121446 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115800    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 20:47:25.121446 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115803    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 20:47:25.121446 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115805    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 20:47:25.121446 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115808    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 20:47:25.121446 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115810    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 20:47:25.121446 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115813    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 20:47:25.121446 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115815    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 20:47:25.121446 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115817    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 20:47:25.121446 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115820    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 20:47:25.121446 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115822    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 20:47:25.121446 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115825    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 20:47:25.121446 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115827    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 20:47:25.121960 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115830    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 20:47:25.121960 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115833    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 20:47:25.121960 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115835    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 20:47:25.121960 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115838    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 20:47:25.121960 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115840    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 20:47:25.121960 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115843    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 20:47:25.121960 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115845    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 20:47:25.121960 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115847    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 20:47:25.121960 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115850    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 20:47:25.121960 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115854    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 20:47:25.121960 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115857    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 20:47:25.121960 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115859    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 20:47:25.121960 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115862    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 20:47:25.121960 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115865    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 20:47:25.121960 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115867    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 20:47:25.121960 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115870    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 20:47:25.121960 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115873    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 20:47:25.121960 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115875    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 20:47:25.121960 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115878    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 20:47:25.122420 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115880    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 20:47:25.122420 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115883    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 20:47:25.122420 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115885    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 20:47:25.122420 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115888    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 20:47:25.122420 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115890    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 20:47:25.122420 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115893    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 20:47:25.122420 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:25.115896    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 20:47:25.122420 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.115901    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 20:47:25.122420 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.116012    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 20:47:25.122420 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.118374    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 20:47:25.122420 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.120132    2565 server.go:1019] "Starting client certificate rotation"
Jul 28 20:47:25.122420 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.120234    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 20:47:25.122420 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.120271    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 20:47:25.123038 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.123012    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 20:47:25.124184 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.124155    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 20:47:25.136317 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.136293    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 28 20:47:25.138993 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.138976    2565 log.go:25] "Validated CRI v1 image API"
Jul 28 20:47:25.139873 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.139856    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 20:47:25.142615 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.142595    2565 fs.go:135] Filesystem UUIDs: map[04fc48d3-60e4-4fcb-8f49-390a6b044db7:/dev/nvme0n1p3 26fc328b-9488-4eb3-8129-a01f06132585:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 28 20:47:25.142688 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.142614    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 20:47:25.149322 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.149204    2565 manager.go:216] Machine: {Timestamp:2026-07-28 20:47:25.147820146 +0000 UTC m=+0.372163124 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098996 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec216d2e5d4cc672750f16ac18416ed0 SystemUUID:ec216d2e-5d4c-c672-750f-16ac18416ed0 BootID:dea4d955-7c18-493f-8364-2995a34ef3a8 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:66:c4:6a:95:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:66:c4:6a:95:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:86:f2:c3:04:1d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 20:47:25.149322 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.149311    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 20:47:25.149466 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.149398    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 20:47:25.150999 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.150982    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 20:47:25.152985 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.152960    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 20:47:25.153164 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.152990    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-170.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 20:47:25.153867 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.153856    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 20:47:25.153900 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.153870    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 20:47:25.153900 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.153883    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 20:47:25.154541 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.154530    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 20:47:25.156424 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.156412    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 20:47:25.161077 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.161065    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 20:47:25.161214 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.161204    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 20:47:25.161290 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.161237    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 20:47:25.161290 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.161252    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 20:47:25.161290 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.161275    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 20:47:25.161290 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.161288    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 28 20:47:25.161424 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.161309    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 20:47:25.161777 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.161765    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 20:47:25.161847 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.161788    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 20:47:25.164287 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.164240    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 20:47:25.166137 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.166117    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 20:47:25.166201 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.166154    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 20:47:25.166911 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.166900    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 20:47:25.166986 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.166917    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 20:47:25.166986 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.166923    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 20:47:25.166986 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.166928    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 20:47:25.166986 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.166936    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 20:47:25.166986 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.166942    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 20:47:25.166986 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.166979    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 20:47:25.166986 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.166985    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 20:47:25.167165 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.166995    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 20:47:25.167165 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.167000    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 20:47:25.167165 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.167015    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 20:47:25.167165 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.167023    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 20:47:25.167907 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.167897    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 20:47:25.167907 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.167906    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 20:47:25.171269 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.171251    2565 server.go:1268] "Started kubelet"
Jul 28 20:47:25.171356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.171337    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 20:47:25.172082 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:25.172051    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-170.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 20:47:25.172222 ip-10-0-134-170 systemd[1]: Started Kubernetes Kubelet.
Jul 28 20:47:25.172398 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:25.172217    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 20:47:25.172398 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.172244    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-170.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 20:47:25.175760 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.171958    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 20:47:25.175844 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.175793    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 20:47:25.177264 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.177244    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jqwq8"
Jul 28 20:47:25.177886 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.177869    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 20:47:25.178139 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.178115    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 20:47:25.178209 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:25.178192    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 20:47:25.182285 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:25.181146    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-170.ec2.internal.18c68f1a3e020d6e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-170.ec2.internal,UID:ip-10-0-134-170.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-170.ec2.internal,},FirstTimestamp:2026-07-28 20:47:25.171223918 +0000 UTC m=+0.395566895,LastTimestamp:2026-07-28 20:47:25.171223918 +0000 UTC m=+0.395566895,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-170.ec2.internal,}"
Jul 28 20:47:25.184382 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.184365    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jqwq8"
Jul 28 20:47:25.191841 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.191828    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 20:47:25.191949 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.191849    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 20:47:25.191995 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:25.191969    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-170.ec2.internal\" not found"
Jul 28 20:47:25.192682 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.192668    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 20:47:25.192775 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.192757    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 20:47:25.192832 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.192780    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 20:47:25.192894 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.192871    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 20:47:25.192894 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.192891    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 20:47:25.193244 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.193205    2565 factory.go:55] Registering systemd factory
Jul 28 20:47:25.193244 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.193220    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 28 20:47:25.193553 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.193538    2565 factory.go:153] Registering CRI-O factory
Jul 28 20:47:25.193655 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.193556    2565 factory.go:223] Registration of the crio container factory successfully
Jul 28 20:47:25.193655 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.193607    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 20:47:25.193655 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.193644    2565 factory.go:103] Registering Raw factory
Jul 28 20:47:25.193803 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.193660    2565 manager.go:1196] Started watching for new ooms in manager
Jul 28 20:47:25.194020 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.194006    2565 manager.go:349] Starting recovery of all containers
Jul 28 20:47:25.205086 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.205068    2565 manager.go:354] Recovery completed
Jul 28 20:47:25.207470 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.207451    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:47:25.209714 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.209701    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:47:25.211469 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.211454    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:47:25.211528 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.211486    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:47:25.211528 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.211496    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:47:25.212012 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.211996    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 20:47:25.212086 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.212013    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 20:47:25.212086 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.212043    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 20:47:25.212147 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.211996    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:47:25.212892 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.212876    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:47:25.212957 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.212902    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:47:25.212957 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.212912    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:47:25.214033 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.214017    2565 policy_none.go:49] "None policy: Start"
Jul 28 20:47:25.214033 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.214035    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 20:47:25.214139 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.214045    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 20:47:25.215844 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:25.215830    2565 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-170.ec2.internal\" not found" node="ip-10-0-134-170.ec2.internal"
Jul 28 20:47:25.216033 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.216022    2565 policy_none.go:47] "Start"
Jul 28 20:47:25.245430 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.245243    2565 manager.go:337] "Starting Device Plugin manager"
Jul 28 20:47:25.245541 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:25.245452    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 20:47:25.245541 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.245465    2565 server.go:87] "Starting device plugin registration server"
Jul 28 20:47:25.245718 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.245706    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 20:47:25.245772 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.245721    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 20:47:25.246332 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.245897    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 20:47:25.246332 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.246007    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 20:47:25.246332 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.246017    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 20:47:25.247083 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:25.246848    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 20:47:25.247083 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:25.246915    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-170.ec2.internal\" not found"
Jul 28 20:47:25.264449 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.264427    2565 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2598 into the system container \"/system.slice\": <nil>"
Jul 28 20:47:25.323821 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.323774    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 20:47:25.326023 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.324910    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 20:47:25.326023 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.324947    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 20:47:25.326023 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.324974    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 20:47:25.326023 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:25.325009    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 20:47:25.329204 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.329183    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:47:25.345849 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.345829    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:47:25.347496 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.347453    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:47:25.347496 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.347483    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:47:25.347496 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.347494    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:47:25.347617 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.347518    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-170.ec2.internal"
Jul 28 20:47:25.358417 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.358398    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-170.ec2.internal"
Jul 28 20:47:25.358468 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:25.358422    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-170.ec2.internal\": node \"ip-10-0-134-170.ec2.internal\" not found"
Jul 28 20:47:25.390073 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:25.390047    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-170.ec2.internal\" not found"
Jul 28 20:47:25.425304 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.425269    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-170.ec2.internal"]
Jul 28 20:47:25.425386 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.425354    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:47:25.426890 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.426874    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:47:25.426954 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.426906    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:47:25.426954 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.426919    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:47:25.429233 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.429221    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:47:25.429361 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.429346    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal"
Jul 28 20:47:25.429455 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.429386    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:47:25.430021 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.430002    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:47:25.430147 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.430032    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:47:25.430147 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.430041    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:47:25.430147 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.430008    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:47:25.430147 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.430084    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:47:25.430147 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.430093    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:47:25.432241 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.432227    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-170.ec2.internal"
Jul 28 20:47:25.432302 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.432253    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 20:47:25.433030 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.433010    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 20:47:25.433113 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.433050    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 20:47:25.433113 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.433066    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeHasSufficientPID"
Jul 28 20:47:25.460431 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:25.460410    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-170.ec2.internal\" not found" node="ip-10-0-134-170.ec2.internal"
Jul 28 20:47:25.463737 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:25.463721    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-170.ec2.internal\" not found" node="ip-10-0-134-170.ec2.internal"
Jul 28 20:47:25.490896 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:25.490874    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-170.ec2.internal\" not found"
Jul 28 20:47:25.493157 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.493143    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/591c0e811a670259632611d3d9fee6d3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal\" (UID: \"591c0e811a670259632611d3d9fee6d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal"
Jul 28 20:47:25.493203 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.493190    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/591c0e811a670259632611d3d9fee6d3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal\" (UID: \"591c0e811a670259632611d3d9fee6d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal"
Jul 28 20:47:25.591794 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:25.591749    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-170.ec2.internal\" not found"
Jul 28 20:47:25.593958 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.593937    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/591c0e811a670259632611d3d9fee6d3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal\" (UID: \"591c0e811a670259632611d3d9fee6d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal"
Jul 28 20:47:25.594014 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.593971    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/591c0e811a670259632611d3d9fee6d3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal\" (UID: \"591c0e811a670259632611d3d9fee6d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal"
Jul 28 20:47:25.594014 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.593996    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29284132b693c2f8453092faf219c32b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-170.ec2.internal\" (UID: \"29284132b693c2f8453092faf219c32b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-170.ec2.internal"
Jul 28 20:47:25.594075 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.594041    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/591c0e811a670259632611d3d9fee6d3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal\" (UID: \"591c0e811a670259632611d3d9fee6d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal"
Jul 28 20:47:25.594075 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.594058    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/591c0e811a670259632611d3d9fee6d3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal\" (UID: \"591c0e811a670259632611d3d9fee6d3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal"
Jul 28 20:47:25.692321 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:25.692280    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-170.ec2.internal\" not found"
Jul 28 20:47:25.694447 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.694431    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29284132b693c2f8453092faf219c32b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-170.ec2.internal\" (UID: \"29284132b693c2f8453092faf219c32b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-170.ec2.internal"
Jul 28 20:47:25.694495 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.694474    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29284132b693c2f8453092faf219c32b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-170.ec2.internal\" (UID: \"29284132b693c2f8453092faf219c32b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-170.ec2.internal"
Jul 28 20:47:25.762580 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.762553    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal"
Jul 28 20:47:25.766376 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:25.766354    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-170.ec2.internal"
Jul 28 20:47:25.793174 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:25.793134    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-170.ec2.internal\" not found"
Jul 28 20:47:25.893463 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:25.893369    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-170.ec2.internal\" not found"
Jul 28 20:47:25.993767 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:25.993731    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-170.ec2.internal\" not found"
Jul 28 20:47:26.094319 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:26.094285    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-170.ec2.internal\" not found"
Jul 28 20:47:26.119556 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:26.119532    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 20:47:26.119982 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:26.119703    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 20:47:26.119982 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:26.119710    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 20:47:26.195079 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:26.195011    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-170.ec2.internal\" not found"
Jul 28 20:47:26.197091 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:26.197065    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 20:42:25 +0000 UTC" deadline="2028-02-16 04:40:58.093693807 +0000 UTC"
Jul 28 20:47:26.197134 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:26.197092    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13615h53m31.896604855s"
Jul 28 20:47:26.295386 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:26.295360    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-170.ec2.internal\" not found"
Jul 28 20:47:26.396034 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:26.396010    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-170.ec2.internal\" not found"
Jul 28 20:47:26.396880 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:26.396851    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29284132b693c2f8453092faf219c32b.slice/crio-1f3b71d8c43bfa94d5dcc1d543d3e93eab2d926680eb9ce35cf9345900ab6806 WatchSource:0}: Error finding container 1f3b71d8c43bfa94d5dcc1d543d3e93eab2d926680eb9ce35cf9345900ab6806: Status 404 returned error can't find the container with id 1f3b71d8c43bfa94d5dcc1d543d3e93eab2d926680eb9ce35cf9345900ab6806
Jul 28 20:47:26.397158 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:26.397145    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591c0e811a670259632611d3d9fee6d3.slice/crio-06b6a7e68576ec66153fa7f1ab3958595b7cce4f05a04434930cde36253eb9ae WatchSource:0}: Error finding container 06b6a7e68576ec66153fa7f1ab3958595b7cce4f05a04434930cde36253eb9ae: Status 404 returned error can't find the container with id 06b6a7e68576ec66153fa7f1ab3958595b7cce4f05a04434930cde36253eb9ae
Jul 28 20:47:26.401453 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:26.401438    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:47:26.436367 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:26.436343    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:47:26.497069 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:26.497005    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-170.ec2.internal\" not found"
Jul 28 20:47:26.522332 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:26.522309    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:47:26.592744 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:26.592720    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-170.ec2.internal"
Jul 28 20:47:26.603967 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:26.603942    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 20:47:26.604696 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:26.604684    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal"
Jul 28 20:47:26.614953 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:26.614934    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 20:47:27.162344 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.162312    2565 apiserver.go:52] "Watching apiserver"
Jul 28 20:47:27.167492 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.167463    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 20:47:27.168955 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.168925    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-170.ec2.internal","openshift-image-registry/node-ca-5ngbd","openshift-multus/multus-additional-cni-plugins-k7vsq","openshift-network-diagnostics/network-check-target-qkd55","openshift-network-operator/iptables-alerter-gkhq2","openshift-ovn-kubernetes/ovnkube-node-g99mf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822","openshift-cluster-node-tuning-operator/tuned-wmxnd","openshift-dns/node-resolver-7prgk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal","openshift-multus/multus-mv4mw","openshift-multus/network-metrics-daemon-jtxnw","kube-system/global-pull-secret-syncer-d8rwp","kube-system/konnectivity-agent-dkngp"]
Jul 28 20:47:27.171942 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.171920    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.174423 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.174398    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zzdfw\""
Jul 28 20:47:27.174970 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.174841    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 20:47:27.174970 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.174888    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 20:47:27.175123 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.175017    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 20:47:27.175406 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.175388    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:27.175505 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:27.175482    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8rwp" podUID="f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22"
Jul 28 20:47:27.178078 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.177841    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.180018 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.179996    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 20:47:27.180018 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.180008    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 20:47:27.180391 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.180373    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 20:47:27.180483 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.180442    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:27.180540 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:27.180518    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qkd55" podUID="ba39afab-1128-4d15-a2ab-bfe072b7eff1"
Jul 28 20:47:27.180595 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.180550    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 20:47:27.180858 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.180838    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8bzpb\""
Jul 28 20:47:27.181110 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.181030    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 20:47:27.182667 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.182440    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gkhq2"
Jul 28 20:47:27.184770 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.184520    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lwhv4\""
Jul 28 20:47:27.184770 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.184612    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 20:47:27.184770 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.184614    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 20:47:27.185060 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.184883    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 20:47:27.185148 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.185132    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.187482 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.187463    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 20:47:27.187617 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.187600    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7prgk"
Jul 28 20:47:27.187710 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.187655    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 20:47:27.187785 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.187771    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 20:47:27.187856 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.187834    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 20:47:27.187940 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.187921    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9nkz8\""
Jul 28 20:47:27.188009 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.187976    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 20:47:27.188303 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.188280    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 20:47:27.189539 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.189519    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 20:47:27.190029 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.190010    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dkngp"
Jul 28 20:47:27.190565 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.190521    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 20:47:27.190773 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.190757    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jk7m9\""
Jul 28 20:47:27.191971 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.191952    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xhbxj\""
Jul 28 20:47:27.192804 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.192104    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 20:47:27.192804 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.192248    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 20:47:27.192804 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.192338    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5ngbd"
Jul 28 20:47:27.193914 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.193887    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 20:47:27.195303 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.195217    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.195406 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.195362    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 20:47:27.195897 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.195816    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2mn7n\""
Jul 28 20:47:27.196255 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.196238    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 20:47:27.196379 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.196361    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 20:47:27.197826 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.197767    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.198882 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.197983    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 20:47:27.198882 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.198261    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7pqxd\""
Jul 28 20:47:27.198882 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.198284    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 20:47:27.200708 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.200093    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 20:47:27.200708 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.200353    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8z4lv\""
Jul 28 20:47:27.200708 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.200583    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:27.200867 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:27.200704    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jtxnw" podUID="6c1be7ce-d683-4505-b781-5f739d0d82b5"
Jul 28 20:47:27.202874 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.202857    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54pxw\" (UniqueName: \"kubernetes.io/projected/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-kube-api-access-54pxw\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.202973 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.202882    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnzhq\" (UniqueName: \"kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq\") pod \"network-check-target-qkd55\" (UID: \"ba39afab-1128-4d15-a2ab-bfe072b7eff1\") " pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:27.202973 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.202901    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-kubelet\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.202973 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.202916    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a967c16a-fcff-43ad-b101-1b5835014439-ovnkube-script-lib\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.202973 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.202931    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b81437ec-1218-4d56-9dc0-5f394a5e45f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.202973 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.202948    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/50035216-1c15-4307-ba45-212599d1e5ed-agent-certs\") pod \"konnectivity-agent-dkngp\" (UID: \"50035216-1c15-4307-ba45-212599d1e5ed\") " pod="kube-system/konnectivity-agent-dkngp"
Jul 28 20:47:27.203227 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.202980    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-var-lib-kubelet\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.203227 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.202998    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-socket-dir\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.203227 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.203013    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-etc-openvswitch\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.203227 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.203031    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-run-ovn-kubernetes\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.203227 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.203045    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b81437ec-1218-4d56-9dc0-5f394a5e45f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.203227 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.203059    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b81437ec-1218-4d56-9dc0-5f394a5e45f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.203227 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.203075    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-sysctl-conf\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.203227 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.203089    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-systemd\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.203227 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.203103    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-var-lib-openvswitch\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.203227 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.203115    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-cni-bin\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.203227 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.203134    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b81437ec-1218-4d56-9dc0-5f394a5e45f4-cnibin\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.203227 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.203155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b81437ec-1218-4d56-9dc0-5f394a5e45f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.203783 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.203269    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppkf7\" (UniqueName: \"kubernetes.io/projected/c20cff94-12ac-4cfe-861f-cb7557a7a0e6-kube-api-access-ppkf7\") pod \"node-ca-5ngbd\" (UID: \"c20cff94-12ac-4cfe-861f-cb7557a7a0e6\") " pod="openshift-image-registry/node-ca-5ngbd"
Jul 28 20:47:27.203783 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.203414    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-registration-dir\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.203884 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.203473    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b35a69ee-6fa5-4373-aadc-51e9aa800ff0-iptables-alerter-script\") pod \"iptables-alerter-gkhq2\" (UID: \"b35a69ee-6fa5-4373-aadc-51e9aa800ff0\") " pod="openshift-network-operator/iptables-alerter-gkhq2"
Jul 28 20:47:27.203938 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.203883    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c20cff94-12ac-4cfe-861f-cb7557a7a0e6-host\") pod \"node-ca-5ngbd\" (UID: \"c20cff94-12ac-4cfe-861f-cb7557a7a0e6\") " pod="openshift-image-registry/node-ca-5ngbd"
Jul 28 20:47:27.203990 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.203942    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.203990 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.203975    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a967c16a-fcff-43ad-b101-1b5835014439-ovn-node-metrics-cert\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.204089 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.204006    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-run-openvswitch\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.204089 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.204030    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret\") pod \"global-pull-secret-syncer-d8rwp\" (UID: \"f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22\") " pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:27.204089 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.204047    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6nbk\" (UniqueName: \"kubernetes.io/projected/b35a69ee-6fa5-4373-aadc-51e9aa800ff0-kube-api-access-l6nbk\") pod \"iptables-alerter-gkhq2\" (UID: \"b35a69ee-6fa5-4373-aadc-51e9aa800ff0\") " pod="openshift-network-operator/iptables-alerter-gkhq2"
Jul 28 20:47:27.204089 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.204076    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-cni-netd\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.204290 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.204096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a967c16a-fcff-43ad-b101-1b5835014439-ovnkube-config\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.204290 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.204114    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a967c16a-fcff-43ad-b101-1b5835014439-env-overrides\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.204290 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.204133    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff2108d7-6551-4eaa-8c51-a234010bc4ba-hosts-file\") pod \"node-resolver-7prgk\" (UID: \"ff2108d7-6551-4eaa-8c51-a234010bc4ba\") " pod="openshift-dns/node-resolver-7prgk"
Jul 28 20:47:27.204290 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.204148    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-sysctl-d\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.204704 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.204683    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 20:47:27.205857 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.204167    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-lib-modules\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.205857 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.205708    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-device-dir\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.205857 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.205781    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-run-ovn\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.205857 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.205815    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.205857 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.205838    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-sys-fs\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.205857 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.205854    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b35a69ee-6fa5-4373-aadc-51e9aa800ff0-host-slash\") pod \"iptables-alerter-gkhq2\" (UID: \"b35a69ee-6fa5-4373-aadc-51e9aa800ff0\") " pod="openshift-network-operator/iptables-alerter-gkhq2"
Jul 28 20:47:27.206175 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.205883    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-run-netns\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.206175 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.205913    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b81437ec-1218-4d56-9dc0-5f394a5e45f4-os-release\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.206175 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.205958    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c20cff94-12ac-4cfe-861f-cb7557a7a0e6-serviceca\") pod \"node-ca-5ngbd\" (UID: \"c20cff94-12ac-4cfe-861f-cb7557a7a0e6\") " pod="openshift-image-registry/node-ca-5ngbd"
Jul 28 20:47:27.206175 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206007    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-host\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.206175 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206126    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wrz6\" (UniqueName: \"kubernetes.io/projected/56d94607-9b1c-45ee-958e-9b66f2d946ea-kube-api-access-6wrz6\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.206175 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206169    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-modprobe-d\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.206478 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206195    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.206478 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206308    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqst8\" (UniqueName: \"kubernetes.io/projected/ff2108d7-6551-4eaa-8c51-a234010bc4ba-kube-api-access-mqst8\") pod \"node-resolver-7prgk\" (UID: \"ff2108d7-6551-4eaa-8c51-a234010bc4ba\") " pod="openshift-dns/node-resolver-7prgk"
Jul 28 20:47:27.206478 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206368    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b81437ec-1218-4d56-9dc0-5f394a5e45f4-system-cni-dir\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.206478 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206397    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-sysconfig\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.206478 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206464    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfc6p\" (UniqueName: \"kubernetes.io/projected/a967c16a-fcff-43ad-b101-1b5835014439-kube-api-access-hfc6p\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.206721 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206501    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-kubelet-config\") pod \"global-pull-secret-syncer-d8rwp\" (UID: \"f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22\") " pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:27.206721 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206540    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-dbus\") pod \"global-pull-secret-syncer-d8rwp\" (UID: \"f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22\") " pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:27.206721 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206565    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-sys\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.206721 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206601    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-tuned\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.206721 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206626    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-log-socket\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.206721 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206675    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-systemd-units\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.206721 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206719    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-run-systemd\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.207021 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206747    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/50035216-1c15-4307-ba45-212599d1e5ed-konnectivity-ca\") pod \"konnectivity-agent-dkngp\" (UID: \"50035216-1c15-4307-ba45-212599d1e5ed\") " pod="kube-system/konnectivity-agent-dkngp"
Jul 28 20:47:27.207021 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206775    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-kubernetes\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.207021 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206807    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-slash\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.207021 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206838    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-node-log\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.207021 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206871    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht7h5\" (UniqueName: \"kubernetes.io/projected/b81437ec-1218-4d56-9dc0-5f394a5e45f4-kube-api-access-ht7h5\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.207021 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206941    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-run\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.207021 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.206969    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56d94607-9b1c-45ee-958e-9b66f2d946ea-tmp\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.207021 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.207018    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2108d7-6551-4eaa-8c51-a234010bc4ba-tmp-dir\") pod \"node-resolver-7prgk\" (UID: \"ff2108d7-6551-4eaa-8c51-a234010bc4ba\") " pod="openshift-dns/node-resolver-7prgk"
Jul 28 20:47:27.210805 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.210785    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:47:27.293483 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.293451    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 20:47:27.307226 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307194    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-modprobe-d\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.307226 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307232    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.307455 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqst8\" (UniqueName: \"kubernetes.io/projected/ff2108d7-6551-4eaa-8c51-a234010bc4ba-kube-api-access-mqst8\") pod \"node-resolver-7prgk\" (UID: \"ff2108d7-6551-4eaa-8c51-a234010bc4ba\") " pod="openshift-dns/node-resolver-7prgk"
Jul 28 20:47:27.307455 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307388    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b81437ec-1218-4d56-9dc0-5f394a5e45f4-system-cni-dir\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.307455 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307415    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-modprobe-d\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.307455 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307445    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.307619 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307462    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b81437ec-1218-4d56-9dc0-5f394a5e45f4-system-cni-dir\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.307619 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307422    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-host-var-lib-cni-bin\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.307619 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-sysconfig\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.307619 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfc6p\" (UniqueName: \"kubernetes.io/projected/a967c16a-fcff-43ad-b101-1b5835014439-kube-api-access-hfc6p\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.307619 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307535    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-kubelet-config\") pod \"global-pull-secret-syncer-d8rwp\" (UID: \"f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22\") " pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:27.307619 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307559    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-dbus\") pod \"global-pull-secret-syncer-d8rwp\" (UID: \"f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22\") " pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:27.307619 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307576    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-sysconfig\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.307952 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307654    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-kubelet-config\") pod \"global-pull-secret-syncer-d8rwp\" (UID: \"f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22\") " pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:27.307952 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307764    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-sys\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.307952 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307800    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-tuned\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.307952 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307766    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-dbus\") pod \"global-pull-secret-syncer-d8rwp\" (UID: \"f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22\") " pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:27.307952 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307857    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-log-socket\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.307952 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307875    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-sys\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.307952 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307892    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-os-release\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.307952 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307920    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgk68\" (UniqueName: \"kubernetes.io/projected/6c1be7ce-d683-4505-b781-5f739d0d82b5-kube-api-access-qgk68\") pod \"network-metrics-daemon-jtxnw\" (UID: \"6c1be7ce-d683-4505-b781-5f739d0d82b5\") " pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:27.307952 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307933    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-log-socket\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.307952 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-systemd-units\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.308434 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307967    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-run-systemd\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.308434 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307982    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/50035216-1c15-4307-ba45-212599d1e5ed-konnectivity-ca\") pod \"konnectivity-agent-dkngp\" (UID: \"50035216-1c15-4307-ba45-212599d1e5ed\") " pod="kube-system/konnectivity-agent-dkngp"
Jul 28 20:47:27.308434 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.307988    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-systemd-units\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.308434 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308005    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-cnibin\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.308434 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308029    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-run-systemd\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.308434 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308029    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-multus-socket-dir-parent\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.308434 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308070    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-kubernetes\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.308434 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308095    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-slash\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.308434 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308118    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-node-log\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.308434 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308118    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 20:47:27.308434 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308164    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht7h5\" (UniqueName: \"kubernetes.io/projected/b81437ec-1218-4d56-9dc0-5f394a5e45f4-kube-api-access-ht7h5\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.308434 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308191    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-run\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.308434 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308221    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56d94607-9b1c-45ee-958e-9b66f2d946ea-tmp\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.308434 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308245    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2108d7-6551-4eaa-8c51-a234010bc4ba-tmp-dir\") pod \"node-resolver-7prgk\" (UID: \"ff2108d7-6551-4eaa-8c51-a234010bc4ba\") " pod="openshift-dns/node-resolver-7prgk"
Jul 28 20:47:27.308434 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308272    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54pxw\" (UniqueName: \"kubernetes.io/projected/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-kube-api-access-54pxw\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.308434 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnzhq\" (UniqueName: \"kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq\") pod \"network-check-target-qkd55\" (UID: \"ba39afab-1128-4d15-a2ab-bfe072b7eff1\") " pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:27.308434 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308329    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-kubelet\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.308434 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308340    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-kubernetes\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.309286 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a967c16a-fcff-43ad-b101-1b5835014439-ovnkube-script-lib\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.309286 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308370    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-slash\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.309286 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308379    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b81437ec-1218-4d56-9dc0-5f394a5e45f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.309286 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308424    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/50035216-1c15-4307-ba45-212599d1e5ed-agent-certs\") pod \"konnectivity-agent-dkngp\" (UID: \"50035216-1c15-4307-ba45-212599d1e5ed\") " pod="kube-system/konnectivity-agent-dkngp"
Jul 28 20:47:27.309286 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308449    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-var-lib-kubelet\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.309286 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-socket-dir\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.309286 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308495    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-etc-openvswitch\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.309286 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308519    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-run-ovn-kubernetes\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.309286 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308543    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b81437ec-1218-4d56-9dc0-5f394a5e45f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.309286 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308568    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b81437ec-1218-4d56-9dc0-5f394a5e45f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.309286 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308577    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/50035216-1c15-4307-ba45-212599d1e5ed-konnectivity-ca\") pod \"konnectivity-agent-dkngp\" (UID: \"50035216-1c15-4307-ba45-212599d1e5ed\") " pod="kube-system/konnectivity-agent-dkngp"
Jul 28 20:47:27.309286 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308917    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-multus-cni-dir\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.309286 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308949    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-multus-conf-dir\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.309286 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308976    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-run\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.309286 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308979    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-sysctl-conf\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.309286 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309005    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2108d7-6551-4eaa-8c51-a234010bc4ba-tmp-dir\") pod \"node-resolver-7prgk\" (UID: \"ff2108d7-6551-4eaa-8c51-a234010bc4ba\") " pod="openshift-dns/node-resolver-7prgk"
Jul 28 20:47:27.309286 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309018    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-systemd\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.310108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309050    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-var-lib-openvswitch\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.310108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309073    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-cni-bin\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.310108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309097    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b81437ec-1218-4d56-9dc0-5f394a5e45f4-cnibin\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.310108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309104    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-sysctl-conf\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.310108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309121    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b81437ec-1218-4d56-9dc0-5f394a5e45f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.310108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309162    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b81437ec-1218-4d56-9dc0-5f394a5e45f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.310108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309181    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-etc-openvswitch\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.310108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309170    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-host-run-multus-certs\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.310108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309239    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-systemd\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.310108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309247    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-var-lib-kubelet\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.310108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309281    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-var-lib-openvswitch\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.310108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309302    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-socket-dir\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.310108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309317    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-cni-bin\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.310108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppkf7\" (UniqueName: \"kubernetes.io/projected/c20cff94-12ac-4cfe-861f-cb7557a7a0e6-kube-api-access-ppkf7\") pod \"node-ca-5ngbd\" (UID: \"c20cff94-12ac-4cfe-861f-cb7557a7a0e6\") " pod="openshift-image-registry/node-ca-5ngbd"
Jul 28 20:47:27.310108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309365    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b81437ec-1218-4d56-9dc0-5f394a5e45f4-cnibin\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.310108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309381    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-registration-dir\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.310108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309408    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-kubelet\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.310895 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309435    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-registration-dir\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.310895 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309455    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b81437ec-1218-4d56-9dc0-5f394a5e45f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.310895 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.308303    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-node-log\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.310895 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309303    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-run-ovn-kubernetes\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.310895 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309702    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b35a69ee-6fa5-4373-aadc-51e9aa800ff0-iptables-alerter-script\") pod \"iptables-alerter-gkhq2\" (UID: \"b35a69ee-6fa5-4373-aadc-51e9aa800ff0\") " pod="openshift-network-operator/iptables-alerter-gkhq2"
Jul 28 20:47:27.310895 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309735    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-system-cni-dir\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.310895 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309759    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-etc-kubernetes\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.310895 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309785    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c20cff94-12ac-4cfe-861f-cb7557a7a0e6-host\") pod \"node-ca-5ngbd\" (UID: \"c20cff94-12ac-4cfe-861f-cb7557a7a0e6\") " pod="openshift-image-registry/node-ca-5ngbd"
Jul 28 20:47:27.310895 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309808    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.310895 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309841    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a967c16a-fcff-43ad-b101-1b5835014439-ovn-node-metrics-cert\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.310895 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309867    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-host-run-netns\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.310895 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309893    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-host-var-lib-kubelet\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.310895 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309897    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b81437ec-1218-4d56-9dc0-5f394a5e45f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.310895 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309918    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb4kv\" (UniqueName: \"kubernetes.io/projected/65544b7e-687b-4ba8-afb4-ff571b8f4939-kube-api-access-tb4kv\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.310895 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309944    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-run-openvswitch\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.310895 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309968    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret\") pod \"global-pull-secret-syncer-d8rwp\" (UID: \"f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22\") " pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:27.310895 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309993    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-hostroot\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.311758 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6nbk\" (UniqueName: \"kubernetes.io/projected/b35a69ee-6fa5-4373-aadc-51e9aa800ff0-kube-api-access-l6nbk\") pod \"iptables-alerter-gkhq2\" (UID: \"b35a69ee-6fa5-4373-aadc-51e9aa800ff0\") " pod="openshift-network-operator/iptables-alerter-gkhq2"
Jul 28 20:47:27.311758 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310043    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-cni-netd\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.311758 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310066    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a967c16a-fcff-43ad-b101-1b5835014439-ovnkube-config\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.311758 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310090    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a967c16a-fcff-43ad-b101-1b5835014439-env-overrides\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.311758 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310109    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b81437ec-1218-4d56-9dc0-5f394a5e45f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.311758 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff2108d7-6551-4eaa-8c51-a234010bc4ba-hosts-file\") pod \"node-resolver-7prgk\" (UID: \"ff2108d7-6551-4eaa-8c51-a234010bc4ba\") " pod="openshift-dns/node-resolver-7prgk"
Jul 28 20:47:27.311758 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310161    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a967c16a-fcff-43ad-b101-1b5835014439-ovnkube-script-lib\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.311758 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310166    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-host-var-lib-cni-multus\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.311758 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310193    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff2108d7-6551-4eaa-8c51-a234010bc4ba-hosts-file\") pod \"node-resolver-7prgk\" (UID: \"ff2108d7-6551-4eaa-8c51-a234010bc4ba\") " pod="openshift-dns/node-resolver-7prgk"
Jul 28 20:47:27.311758 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.309968    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c20cff94-12ac-4cfe-861f-cb7557a7a0e6-host\") pod \"node-ca-5ngbd\" (UID: \"c20cff94-12ac-4cfe-861f-cb7557a7a0e6\") " pod="openshift-image-registry/node-ca-5ngbd"
Jul 28 20:47:27.311758 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:27.310273    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:27.311758 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310280    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-run-openvswitch\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.311758 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310331    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-cni-netd\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.311758 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:27.310349    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret podName:f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:27.810327881 +0000 UTC m=+3.034671067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret") pod "global-pull-secret-syncer-d8rwp" (UID: "f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:27.311758 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310441    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b35a69ee-6fa5-4373-aadc-51e9aa800ff0-iptables-alerter-script\") pod \"iptables-alerter-gkhq2\" (UID: \"b35a69ee-6fa5-4373-aadc-51e9aa800ff0\") " pod="openshift-network-operator/iptables-alerter-gkhq2"
Jul 28 20:47:27.311758 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310510    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.311758 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310441    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs\") pod \"network-metrics-daemon-jtxnw\" (UID: \"6c1be7ce-d683-4505-b781-5f739d0d82b5\") " pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:27.312470 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310559    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-sysctl-d\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.312470 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310651    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-lib-modules\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.312470 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310687    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-device-dir\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.312470 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310739    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-run-ovn\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.312470 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310766    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.312470 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310795    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-host-run-k8s-cni-cncf-io\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.312470 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310812    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a967c16a-fcff-43ad-b101-1b5835014439-ovnkube-config\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.312470 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310816    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a967c16a-fcff-43ad-b101-1b5835014439-env-overrides\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.312470 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310847    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-device-dir\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.312470 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-sys-fs\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.312470 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310888    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.312470 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310924    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b35a69ee-6fa5-4373-aadc-51e9aa800ff0-host-slash\") pod \"iptables-alerter-gkhq2\" (UID: \"b35a69ee-6fa5-4373-aadc-51e9aa800ff0\") " pod="openshift-network-operator/iptables-alerter-gkhq2"
Jul 28 20:47:27.312470 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310951    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-run-netns\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.312470 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310977    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b81437ec-1218-4d56-9dc0-5f394a5e45f4-os-release\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.312470 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.310926    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-run-ovn\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.312470 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.311003    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-sysctl-d\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.312470 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.311037    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/65544b7e-687b-4ba8-afb4-ff571b8f4939-multus-daemon-config\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.313333 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.311063    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b81437ec-1218-4d56-9dc0-5f394a5e45f4-os-release\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.313333 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.311091    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c20cff94-12ac-4cfe-861f-cb7557a7a0e6-serviceca\") pod \"node-ca-5ngbd\" (UID: \"c20cff94-12ac-4cfe-861f-cb7557a7a0e6\") " pod="openshift-image-registry/node-ca-5ngbd"
Jul 28 20:47:27.313333 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.311135    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a967c16a-fcff-43ad-b101-1b5835014439-host-run-netns\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.313333 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.311144    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-lib-modules\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.313333 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.311145    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-sys-fs\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.313333 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.311182    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-host\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.313333 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.311184    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b35a69ee-6fa5-4373-aadc-51e9aa800ff0-host-slash\") pod \"iptables-alerter-gkhq2\" (UID: \"b35a69ee-6fa5-4373-aadc-51e9aa800ff0\") " pod="openshift-network-operator/iptables-alerter-gkhq2"
Jul 28 20:47:27.313333 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.311233    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wrz6\" (UniqueName: \"kubernetes.io/projected/56d94607-9b1c-45ee-958e-9b66f2d946ea-kube-api-access-6wrz6\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.313333 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.311240    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/56d94607-9b1c-45ee-958e-9b66f2d946ea-host\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.313333 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.311276    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65544b7e-687b-4ba8-afb4-ff571b8f4939-cni-binary-copy\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.313333 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.311576    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c20cff94-12ac-4cfe-861f-cb7557a7a0e6-serviceca\") pod \"node-ca-5ngbd\" (UID: \"c20cff94-12ac-4cfe-861f-cb7557a7a0e6\") " pod="openshift-image-registry/node-ca-5ngbd"
Jul 28 20:47:27.313333 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.312754    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/56d94607-9b1c-45ee-958e-9b66f2d946ea-etc-tuned\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.313333 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.313192    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56d94607-9b1c-45ee-958e-9b66f2d946ea-tmp\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.313949 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.313455    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/50035216-1c15-4307-ba45-212599d1e5ed-agent-certs\") pod \"konnectivity-agent-dkngp\" (UID: \"50035216-1c15-4307-ba45-212599d1e5ed\") " pod="kube-system/konnectivity-agent-dkngp"
Jul 28 20:47:27.315283 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:27.315138    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:47:27.315283 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.315156    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a967c16a-fcff-43ad-b101-1b5835014439-ovn-node-metrics-cert\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.315283 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:27.315170    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:47:27.315283 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:27.315185    2565 projected.go:196] Error preparing data for projected volume kube-api-access-fnzhq for pod openshift-network-diagnostics/network-check-target-qkd55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:27.315547 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:27.315316    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq podName:ba39afab-1128-4d15-a2ab-bfe072b7eff1 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:27.815273993 +0000 UTC m=+3.039616977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fnzhq" (UniqueName: "kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq") pod "network-check-target-qkd55" (UID: "ba39afab-1128-4d15-a2ab-bfe072b7eff1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:27.317477 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.317453    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54pxw\" (UniqueName: \"kubernetes.io/projected/7b99cc53-e3f9-4970-a93a-1a052da4dd9d-kube-api-access-54pxw\") pod \"aws-ebs-csi-driver-node-xs822\" (UID: \"7b99cc53-e3f9-4970-a93a-1a052da4dd9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.318286 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.318265    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfc6p\" (UniqueName: \"kubernetes.io/projected/a967c16a-fcff-43ad-b101-1b5835014439-kube-api-access-hfc6p\") pod \"ovnkube-node-g99mf\" (UID: \"a967c16a-fcff-43ad-b101-1b5835014439\") " pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.318737 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.318681    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppkf7\" (UniqueName: \"kubernetes.io/projected/c20cff94-12ac-4cfe-861f-cb7557a7a0e6-kube-api-access-ppkf7\") pod \"node-ca-5ngbd\" (UID: \"c20cff94-12ac-4cfe-861f-cb7557a7a0e6\") " pod="openshift-image-registry/node-ca-5ngbd"
Jul 28 20:47:27.319543 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.319512    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht7h5\" (UniqueName: \"kubernetes.io/projected/b81437ec-1218-4d56-9dc0-5f394a5e45f4-kube-api-access-ht7h5\") pod \"multus-additional-cni-plugins-k7vsq\" (UID: \"b81437ec-1218-4d56-9dc0-5f394a5e45f4\") " pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.319630 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.319597    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqst8\" (UniqueName: \"kubernetes.io/projected/ff2108d7-6551-4eaa-8c51-a234010bc4ba-kube-api-access-mqst8\") pod \"node-resolver-7prgk\" (UID: \"ff2108d7-6551-4eaa-8c51-a234010bc4ba\") " pod="openshift-dns/node-resolver-7prgk"
Jul 28 20:47:27.324173 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.324149    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6nbk\" (UniqueName: \"kubernetes.io/projected/b35a69ee-6fa5-4373-aadc-51e9aa800ff0-kube-api-access-l6nbk\") pod \"iptables-alerter-gkhq2\" (UID: \"b35a69ee-6fa5-4373-aadc-51e9aa800ff0\") " pod="openshift-network-operator/iptables-alerter-gkhq2"
Jul 28 20:47:27.327083 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.327055    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wrz6\" (UniqueName: \"kubernetes.io/projected/56d94607-9b1c-45ee-958e-9b66f2d946ea-kube-api-access-6wrz6\") pod \"tuned-wmxnd\" (UID: \"56d94607-9b1c-45ee-958e-9b66f2d946ea\") " pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.330884 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.330810    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal" event={"ID":"591c0e811a670259632611d3d9fee6d3","Type":"ContainerStarted","Data":"06b6a7e68576ec66153fa7f1ab3958595b7cce4f05a04434930cde36253eb9ae"}
Jul 28 20:47:27.331330 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.331223    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9gxpm"
Jul 28 20:47:27.332311 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.332285    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-170.ec2.internal" event={"ID":"29284132b693c2f8453092faf219c32b","Type":"ContainerStarted","Data":"1f3b71d8c43bfa94d5dcc1d543d3e93eab2d926680eb9ce35cf9345900ab6806"}
Jul 28 20:47:27.340386 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.340361    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9gxpm"
Jul 28 20:47:27.411953 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.411862    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-cnibin\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.411953 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.411925    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-multus-socket-dir-parent\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.411953 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.411961    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-multus-cni-dir\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.412269 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.411984    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-multus-conf-dir\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.412269 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.411994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-cnibin\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.412269 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412011    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-host-run-multus-certs\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.412269 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412054    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-host-run-multus-certs\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.412269 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412055    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-system-cni-dir\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.412269 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412061    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-multus-socket-dir-parent\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.412269 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412061    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-multus-cni-dir\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.412269 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412081    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-etc-kubernetes\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.412269 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412105    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-multus-conf-dir\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.412269 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412114    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-system-cni-dir\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.412269 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412152    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-etc-kubernetes\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.412269 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412199    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-host-run-netns\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.412269 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412251    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-host-var-lib-kubelet\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.412269 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412270    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-host-var-lib-kubelet\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.413007 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412316    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-host-run-netns\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.413007 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412348    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb4kv\" (UniqueName: \"kubernetes.io/projected/65544b7e-687b-4ba8-afb4-ff571b8f4939-kube-api-access-tb4kv\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.413007 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412401    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-hostroot\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.413007 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412428    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-host-var-lib-cni-multus\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.413007 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412451    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs\") pod \"network-metrics-daemon-jtxnw\" (UID: \"6c1be7ce-d683-4505-b781-5f739d0d82b5\") " pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:27.413007 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412505    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-host-run-k8s-cni-cncf-io\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.413007 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412521    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-hostroot\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.413007 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412535    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/65544b7e-687b-4ba8-afb4-ff571b8f4939-multus-daemon-config\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.413007 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412570    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-host-run-k8s-cni-cncf-io\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.413007 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412526    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-host-var-lib-cni-multus\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.413007 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412572    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65544b7e-687b-4ba8-afb4-ff571b8f4939-cni-binary-copy\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.413007 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412619    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-host-var-lib-cni-bin\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.413007 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:27.412613    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:27.413007 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412666    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-os-release\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.413007 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412686    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-host-var-lib-cni-bin\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.413007 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:27.412699    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs podName:6c1be7ce-d683-4505-b781-5f739d0d82b5 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:27.912678418 +0000 UTC m=+3.137021386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs") pod "network-metrics-daemon-jtxnw" (UID: "6c1be7ce-d683-4505-b781-5f739d0d82b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:27.413007 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgk68\" (UniqueName: \"kubernetes.io/projected/6c1be7ce-d683-4505-b781-5f739d0d82b5-kube-api-access-qgk68\") pod \"network-metrics-daemon-jtxnw\" (UID: \"6c1be7ce-d683-4505-b781-5f739d0d82b5\") " pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:27.413007 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.412728    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/65544b7e-687b-4ba8-afb4-ff571b8f4939-os-release\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.413754 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.413095    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/65544b7e-687b-4ba8-afb4-ff571b8f4939-multus-daemon-config\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.413754 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.413102    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65544b7e-687b-4ba8-afb4-ff571b8f4939-cni-binary-copy\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.421867 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.421841    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb4kv\" (UniqueName: \"kubernetes.io/projected/65544b7e-687b-4ba8-afb4-ff571b8f4939-kube-api-access-tb4kv\") pod \"multus-mv4mw\" (UID: \"65544b7e-687b-4ba8-afb4-ff571b8f4939\") " pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.421987 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.421944    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgk68\" (UniqueName: \"kubernetes.io/projected/6c1be7ce-d683-4505-b781-5f739d0d82b5-kube-api-access-qgk68\") pod \"network-metrics-daemon-jtxnw\" (UID: \"6c1be7ce-d683-4505-b781-5f739d0d82b5\") " pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:27.486847 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.486807    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822"
Jul 28 20:47:27.495214 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.495190    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k7vsq"
Jul 28 20:47:27.508084 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.508059    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gkhq2"
Jul 28 20:47:27.514866 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.514845    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:27.523530 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.523505    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7prgk"
Jul 28 20:47:27.530278 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.530254    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dkngp"
Jul 28 20:47:27.537775 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.537754    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5ngbd"
Jul 28 20:47:27.545560 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.545538    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wmxnd"
Jul 28 20:47:27.551127 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.551102    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mv4mw"
Jul 28 20:47:27.658473 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.658437    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 20:47:27.815361 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.815261    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret\") pod \"global-pull-secret-syncer-d8rwp\" (UID: \"f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22\") " pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:27.815519 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:27.815434    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:27.815519 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:27.815506    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret podName:f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:28.815490236 +0000 UTC m=+4.039833200 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret") pod "global-pull-secret-syncer-d8rwp" (UID: "f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:27.916403 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.916367    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs\") pod \"network-metrics-daemon-jtxnw\" (UID: \"6c1be7ce-d683-4505-b781-5f739d0d82b5\") " pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:27.916587 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:27.916437    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnzhq\" (UniqueName: \"kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq\") pod \"network-check-target-qkd55\" (UID: \"ba39afab-1128-4d15-a2ab-bfe072b7eff1\") " pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:27.916587 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:27.916551    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:27.916728 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:27.916626    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs podName:6c1be7ce-d683-4505-b781-5f739d0d82b5 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:28.916604181 +0000 UTC m=+4.140947169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs") pod "network-metrics-daemon-jtxnw" (UID: "6c1be7ce-d683-4505-b781-5f739d0d82b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:27.916728 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:27.916556    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:47:27.916728 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:27.916670    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:47:27.916728 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:27.916681    2565 projected.go:196] Error preparing data for projected volume kube-api-access-fnzhq for pod openshift-network-diagnostics/network-check-target-qkd55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:27.916728 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:27.916724    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq podName:ba39afab-1128-4d15-a2ab-bfe072b7eff1 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:28.916711568 +0000 UTC m=+4.141054547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fnzhq" (UniqueName: "kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq") pod "network-check-target-qkd55" (UID: "ba39afab-1128-4d15-a2ab-bfe072b7eff1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:28.207871 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:28.207838    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b99cc53_e3f9_4970_a93a_1a052da4dd9d.slice/crio-999dd0264e2ce222d3210ed8b207d4e74a3f8d4795b0acd98641b286e11c621c WatchSource:0}: Error finding container 999dd0264e2ce222d3210ed8b207d4e74a3f8d4795b0acd98641b286e11c621c: Status 404 returned error can't find the container with id 999dd0264e2ce222d3210ed8b207d4e74a3f8d4795b0acd98641b286e11c621c
Jul 28 20:47:28.210423 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:28.210311    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2108d7_6551_4eaa_8c51_a234010bc4ba.slice/crio-cc238e921b44f50bfb5603c1d15a157767a85fcdb8cd0d2a41fd4644e980cbfe WatchSource:0}: Error finding container cc238e921b44f50bfb5603c1d15a157767a85fcdb8cd0d2a41fd4644e980cbfe: Status 404 returned error can't find the container with id cc238e921b44f50bfb5603c1d15a157767a85fcdb8cd0d2a41fd4644e980cbfe
Jul 28 20:47:28.211786 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:28.211761    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81437ec_1218_4d56_9dc0_5f394a5e45f4.slice/crio-59d263cd8a47fe90f6b026e5ef0f90a868ac5422a19961cdcb69b513ef8b4cf5 WatchSource:0}: Error finding container 59d263cd8a47fe90f6b026e5ef0f90a868ac5422a19961cdcb69b513ef8b4cf5: Status 404 returned error can't find the container with id 59d263cd8a47fe90f6b026e5ef0f90a868ac5422a19961cdcb69b513ef8b4cf5
Jul 28 20:47:28.233320 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:28.233239    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc20cff94_12ac_4cfe_861f_cb7557a7a0e6.slice/crio-04d703c60ec47f138e91aa8555c844c63e9a6d9d562d176f88ef0ba02b6964b7 WatchSource:0}: Error finding container 04d703c60ec47f138e91aa8555c844c63e9a6d9d562d176f88ef0ba02b6964b7: Status 404 returned error can't find the container with id 04d703c60ec47f138e91aa8555c844c63e9a6d9d562d176f88ef0ba02b6964b7
Jul 28 20:47:28.234817 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:28.234791    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65544b7e_687b_4ba8_afb4_ff571b8f4939.slice/crio-e8b942d275489db88513b88857efae114fea34d749f55090f829db156a175539 WatchSource:0}: Error finding container e8b942d275489db88513b88857efae114fea34d749f55090f829db156a175539: Status 404 returned error can't find the container with id e8b942d275489db88513b88857efae114fea34d749f55090f829db156a175539
Jul 28 20:47:28.235554 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:28.235532    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda967c16a_fcff_43ad_b101_1b5835014439.slice/crio-2499d0baf2c06bb58f379498b048ec63c040b775ad723212a2d93a4fb0e20acc WatchSource:0}: Error finding container 2499d0baf2c06bb58f379498b048ec63c040b775ad723212a2d93a4fb0e20acc: Status 404 returned error can't find the container with id 2499d0baf2c06bb58f379498b048ec63c040b775ad723212a2d93a4fb0e20acc
Jul 28 20:47:28.236556 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:28.236540    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d94607_9b1c_45ee_958e_9b66f2d946ea.slice/crio-4f68ecd431126b27df701766917533515f4aa880f385d648cb54aae13516188d WatchSource:0}: Error finding container 4f68ecd431126b27df701766917533515f4aa880f385d648cb54aae13516188d: Status 404 returned error can't find the container with id 4f68ecd431126b27df701766917533515f4aa880f385d648cb54aae13516188d
Jul 28 20:47:28.238418 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:28.238269    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb35a69ee_6fa5_4373_aadc_51e9aa800ff0.slice/crio-50dcfdfdc998863f3567900e644f0d228c20bb413d4153ce54af451190c24e1d WatchSource:0}: Error finding container 50dcfdfdc998863f3567900e644f0d228c20bb413d4153ce54af451190c24e1d: Status 404 returned error can't find the container with id 50dcfdfdc998863f3567900e644f0d228c20bb413d4153ce54af451190c24e1d
Jul 28 20:47:28.239451 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:47:28.239429    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50035216_1c15_4307_ba45_212599d1e5ed.slice/crio-75110f1547a31cd8813abc23d2b2fafc44005b8081bc71954769d0c8115e311d WatchSource:0}: Error finding container 75110f1547a31cd8813abc23d2b2fafc44005b8081bc71954769d0c8115e311d: Status 404 returned error can't find the container with id 75110f1547a31cd8813abc23d2b2fafc44005b8081bc71954769d0c8115e311d
Jul 28 20:47:28.325772 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:28.325565    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:28.325884 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:28.325864    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8rwp" podUID="f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22"
Jul 28 20:47:28.335215 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:28.335178    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gkhq2" event={"ID":"b35a69ee-6fa5-4373-aadc-51e9aa800ff0","Type":"ContainerStarted","Data":"50dcfdfdc998863f3567900e644f0d228c20bb413d4153ce54af451190c24e1d"}
Jul 28 20:47:28.336511 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:28.336484    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g99mf" event={"ID":"a967c16a-fcff-43ad-b101-1b5835014439","Type":"ContainerStarted","Data":"2499d0baf2c06bb58f379498b048ec63c040b775ad723212a2d93a4fb0e20acc"}
Jul 28 20:47:28.337694 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:28.337649    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mv4mw" event={"ID":"65544b7e-687b-4ba8-afb4-ff571b8f4939","Type":"ContainerStarted","Data":"e8b942d275489db88513b88857efae114fea34d749f55090f829db156a175539"}
Jul 28 20:47:28.338701 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:28.338675    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5ngbd" event={"ID":"c20cff94-12ac-4cfe-861f-cb7557a7a0e6","Type":"ContainerStarted","Data":"04d703c60ec47f138e91aa8555c844c63e9a6d9d562d176f88ef0ba02b6964b7"}
Jul 28 20:47:28.339797 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:28.339775    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822" event={"ID":"7b99cc53-e3f9-4970-a93a-1a052da4dd9d","Type":"ContainerStarted","Data":"999dd0264e2ce222d3210ed8b207d4e74a3f8d4795b0acd98641b286e11c621c"}
Jul 28 20:47:28.341746 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:28.341723    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-170.ec2.internal" event={"ID":"29284132b693c2f8453092faf219c32b","Type":"ContainerStarted","Data":"93646ab08c873443a7dc1083d9d11a5e5db9901232add377c85471c9158c2afe"}
Jul 28 20:47:28.342844 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:28.342817    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dkngp" event={"ID":"50035216-1c15-4307-ba45-212599d1e5ed","Type":"ContainerStarted","Data":"75110f1547a31cd8813abc23d2b2fafc44005b8081bc71954769d0c8115e311d"}
Jul 28 20:47:28.343966 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:28.343946    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wmxnd" event={"ID":"56d94607-9b1c-45ee-958e-9b66f2d946ea","Type":"ContainerStarted","Data":"4f68ecd431126b27df701766917533515f4aa880f385d648cb54aae13516188d"}
Jul 28 20:47:28.344985 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:28.344962    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 20:42:27 +0000 UTC" deadline="2028-03-27 15:42:17.917161099 +0000 UTC"
Jul 28 20:47:28.344985 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:28.344984    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14586h54m49.572180054s"
Jul 28 20:47:28.345125 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:28.345111    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7vsq" event={"ID":"b81437ec-1218-4d56-9dc0-5f394a5e45f4","Type":"ContainerStarted","Data":"59d263cd8a47fe90f6b026e5ef0f90a868ac5422a19961cdcb69b513ef8b4cf5"}
Jul 28 20:47:28.347030 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:28.347006    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7prgk" event={"ID":"ff2108d7-6551-4eaa-8c51-a234010bc4ba","Type":"ContainerStarted","Data":"cc238e921b44f50bfb5603c1d15a157767a85fcdb8cd0d2a41fd4644e980cbfe"}
Jul 28 20:47:28.356247 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:28.356192    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-170.ec2.internal" podStartSLOduration=2.356177662 podStartE2EDuration="2.356177662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:28.355630746 +0000 UTC m=+3.579973732" watchObservedRunningTime="2026-07-28 20:47:28.356177662 +0000 UTC m=+3.580520648"
Jul 28 20:47:28.823412 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:28.823323    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret\") pod \"global-pull-secret-syncer-d8rwp\" (UID: \"f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22\") " pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:28.823580 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:28.823529    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:28.823664 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:28.823598    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret podName:f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:30.823577228 +0000 UTC m=+6.047920218 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret") pod "global-pull-secret-syncer-d8rwp" (UID: "f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:28.924516 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:28.924335    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnzhq\" (UniqueName: \"kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq\") pod \"network-check-target-qkd55\" (UID: \"ba39afab-1128-4d15-a2ab-bfe072b7eff1\") " pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:28.924516 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:28.924416    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs\") pod \"network-metrics-daemon-jtxnw\" (UID: \"6c1be7ce-d683-4505-b781-5f739d0d82b5\") " pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:28.924775 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:28.924568    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:28.924775 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:28.924646    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs podName:6c1be7ce-d683-4505-b781-5f739d0d82b5 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:30.924613216 +0000 UTC m=+6.148956186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs") pod "network-metrics-daemon-jtxnw" (UID: "6c1be7ce-d683-4505-b781-5f739d0d82b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:28.925277 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:28.925054    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:47:28.925277 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:28.925073    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:47:28.925277 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:28.925086    2565 projected.go:196] Error preparing data for projected volume kube-api-access-fnzhq for pod openshift-network-diagnostics/network-check-target-qkd55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:28.925277 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:28.925129    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq podName:ba39afab-1128-4d15-a2ab-bfe072b7eff1 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:30.925114322 +0000 UTC m=+6.149457295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fnzhq" (UniqueName: "kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq") pod "network-check-target-qkd55" (UID: "ba39afab-1128-4d15-a2ab-bfe072b7eff1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:29.333420 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:29.333387    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:29.334048 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:29.333581    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jtxnw" podUID="6c1be7ce-d683-4505-b781-5f739d0d82b5"
Jul 28 20:47:29.334170 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:29.334148    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:29.334307 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:29.334287    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qkd55" podUID="ba39afab-1128-4d15-a2ab-bfe072b7eff1"
Jul 28 20:47:29.346566 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:29.346495    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 20:42:27 +0000 UTC" deadline="2028-01-23 00:12:22.275702084 +0000 UTC"
Jul 28 20:47:29.346566 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:29.346545    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13035h24m52.929161287s"
Jul 28 20:47:29.375784 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:29.375744    2565 generic.go:356] "Generic (PLEG): container finished" podID="591c0e811a670259632611d3d9fee6d3" containerID="03ef8d523f8562bb461c0bb5900ba82025836e779d43135331c8dd049f86ab24" exitCode=0
Jul 28 20:47:29.375991 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:29.375965    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal" event={"ID":"591c0e811a670259632611d3d9fee6d3","Type":"ContainerDied","Data":"03ef8d523f8562bb461c0bb5900ba82025836e779d43135331c8dd049f86ab24"}
Jul 28 20:47:30.326241 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:30.326160    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:30.326404 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:30.326300    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8rwp" podUID="f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22"
Jul 28 20:47:30.400689 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:30.400599    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal" event={"ID":"591c0e811a670259632611d3d9fee6d3","Type":"ContainerStarted","Data":"71f8dfb6b0b551480f18e7cc6d52fe9eb9e13f602d4e042d15e3f252a2fde392"}
Jul 28 20:47:30.847193 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:30.846532    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret\") pod \"global-pull-secret-syncer-d8rwp\" (UID: \"f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22\") " pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:30.847193 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:30.846738    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:30.847193 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:30.846802    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret podName:f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:34.846784044 +0000 UTC m=+10.071127025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret") pod "global-pull-secret-syncer-d8rwp" (UID: "f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:30.947532 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:30.947491    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnzhq\" (UniqueName: \"kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq\") pod \"network-check-target-qkd55\" (UID: \"ba39afab-1128-4d15-a2ab-bfe072b7eff1\") " pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:30.947746 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:30.947626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs\") pod \"network-metrics-daemon-jtxnw\" (UID: \"6c1be7ce-d683-4505-b781-5f739d0d82b5\") " pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:30.947838 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:30.947817    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:30.947902 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:30.947892    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs podName:6c1be7ce-d683-4505-b781-5f739d0d82b5 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:34.947872774 +0000 UTC m=+10.172215740 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs") pod "network-metrics-daemon-jtxnw" (UID: "6c1be7ce-d683-4505-b781-5f739d0d82b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:30.948394 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:30.948374    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:47:30.948507 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:30.948400    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:47:30.948507 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:30.948412    2565 projected.go:196] Error preparing data for projected volume kube-api-access-fnzhq for pod openshift-network-diagnostics/network-check-target-qkd55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:30.948507 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:30.948458    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq podName:ba39afab-1128-4d15-a2ab-bfe072b7eff1 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:34.948441912 +0000 UTC m=+10.172784876 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fnzhq" (UniqueName: "kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq") pod "network-check-target-qkd55" (UID: "ba39afab-1128-4d15-a2ab-bfe072b7eff1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:31.325616 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:31.325536    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:31.325790 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:31.325699    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jtxnw" podUID="6c1be7ce-d683-4505-b781-5f739d0d82b5"
Jul 28 20:47:31.326107 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:31.326085    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:31.326212 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:31.326188    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qkd55" podUID="ba39afab-1128-4d15-a2ab-bfe072b7eff1"
Jul 28 20:47:32.326021 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:32.325984    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:32.326493 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:32.326131    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8rwp" podUID="f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22"
Jul 28 20:47:33.326130 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:33.326095    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:33.326571 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:33.326095    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:33.326571 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:33.326250    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jtxnw" podUID="6c1be7ce-d683-4505-b781-5f739d0d82b5"
Jul 28 20:47:33.326571 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:33.326315    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qkd55" podUID="ba39afab-1128-4d15-a2ab-bfe072b7eff1"
Jul 28 20:47:34.325606 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:34.325564    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:34.325793 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:34.325731    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8rwp" podUID="f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22"
Jul 28 20:47:34.879222 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:34.879183    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret\") pod \"global-pull-secret-syncer-d8rwp\" (UID: \"f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22\") " pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:34.879716 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:34.879344    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:34.879716 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:34.879425    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret podName:f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:42.879401719 +0000 UTC m=+18.103744685 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret") pod "global-pull-secret-syncer-d8rwp" (UID: "f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:34.980555 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:34.979904    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnzhq\" (UniqueName: \"kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq\") pod \"network-check-target-qkd55\" (UID: \"ba39afab-1128-4d15-a2ab-bfe072b7eff1\") " pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:34.980555 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:34.979998    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs\") pod \"network-metrics-daemon-jtxnw\" (UID: \"6c1be7ce-d683-4505-b781-5f739d0d82b5\") " pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:34.980555 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:34.980108    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:34.980555 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:34.980109    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:47:34.980555 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:34.980140    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:47:34.980555 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:34.980153    2565 projected.go:196] Error preparing data for projected volume kube-api-access-fnzhq for pod openshift-network-diagnostics/network-check-target-qkd55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:34.980555 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:34.980178    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs podName:6c1be7ce-d683-4505-b781-5f739d0d82b5 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:42.980157707 +0000 UTC m=+18.204500687 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs") pod "network-metrics-daemon-jtxnw" (UID: "6c1be7ce-d683-4505-b781-5f739d0d82b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:34.980555 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:34.980205    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq podName:ba39afab-1128-4d15-a2ab-bfe072b7eff1 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:42.980189637 +0000 UTC m=+18.204532605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fnzhq" (UniqueName: "kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq") pod "network-check-target-qkd55" (UID: "ba39afab-1128-4d15-a2ab-bfe072b7eff1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:35.327450 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:35.326754    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:35.327450 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:35.326881    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jtxnw" podUID="6c1be7ce-d683-4505-b781-5f739d0d82b5"
Jul 28 20:47:35.327450 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:35.327260    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:35.327450 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:35.327354    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qkd55" podUID="ba39afab-1128-4d15-a2ab-bfe072b7eff1"
Jul 28 20:47:36.325332 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:36.325289    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:36.325879 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:36.325505    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8rwp" podUID="f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22"
Jul 28 20:47:37.325782 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:37.325738    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:37.326287 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:37.325875    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qkd55" podUID="ba39afab-1128-4d15-a2ab-bfe072b7eff1"
Jul 28 20:47:37.326287 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:37.326181    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:37.326410 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:37.326302    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jtxnw" podUID="6c1be7ce-d683-4505-b781-5f739d0d82b5"
Jul 28 20:47:38.325330 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:38.325295    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:38.325497 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:38.325419    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8rwp" podUID="f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22"
Jul 28 20:47:39.325735 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:39.325657    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:39.326161 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:39.325657    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:39.326161 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:39.325795    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jtxnw" podUID="6c1be7ce-d683-4505-b781-5f739d0d82b5"
Jul 28 20:47:39.326161 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:39.325870    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qkd55" podUID="ba39afab-1128-4d15-a2ab-bfe072b7eff1"
Jul 28 20:47:40.326126 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:40.326077    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:40.326629 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:40.326221    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8rwp" podUID="f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22"
Jul 28 20:47:41.325497 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:41.325455    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:41.325497 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:41.325479    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:41.325744 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:41.325618    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jtxnw" podUID="6c1be7ce-d683-4505-b781-5f739d0d82b5"
Jul 28 20:47:41.325744 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:41.325718    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qkd55" podUID="ba39afab-1128-4d15-a2ab-bfe072b7eff1"
Jul 28 20:47:42.325356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:42.325313    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:42.325831 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:42.325439    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8rwp" podUID="f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22"
Jul 28 20:47:42.939818 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:42.939782    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret\") pod \"global-pull-secret-syncer-d8rwp\" (UID: \"f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22\") " pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:42.940078 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:42.939951    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:42.940078 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:42.940035    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret podName:f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:58.94001464 +0000 UTC m=+34.164357621 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret") pod "global-pull-secret-syncer-d8rwp" (UID: "f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:43.040288 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:43.040251    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs\") pod \"network-metrics-daemon-jtxnw\" (UID: \"6c1be7ce-d683-4505-b781-5f739d0d82b5\") " pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:43.040459 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:43.040311    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnzhq\" (UniqueName: \"kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq\") pod \"network-check-target-qkd55\" (UID: \"ba39afab-1128-4d15-a2ab-bfe072b7eff1\") " pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:43.040459 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:43.040414    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:43.040546 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:43.040496    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs podName:6c1be7ce-d683-4505-b781-5f739d0d82b5 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:59.040473646 +0000 UTC m=+34.264816612 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs") pod "network-metrics-daemon-jtxnw" (UID: "6c1be7ce-d683-4505-b781-5f739d0d82b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:43.040546 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:43.040422    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:47:43.040546 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:43.040539    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:47:43.040696 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:43.040555    2565 projected.go:196] Error preparing data for projected volume kube-api-access-fnzhq for pod openshift-network-diagnostics/network-check-target-qkd55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:43.040696 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:43.040613    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq podName:ba39afab-1128-4d15-a2ab-bfe072b7eff1 nodeName:}" failed. No retries permitted until 2026-07-28 20:47:59.040598149 +0000 UTC m=+34.264941132 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fnzhq" (UniqueName: "kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq") pod "network-check-target-qkd55" (UID: "ba39afab-1128-4d15-a2ab-bfe072b7eff1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:43.325807 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:43.325730    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:43.326251 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:43.325731    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:43.326251 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:43.325856    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qkd55" podUID="ba39afab-1128-4d15-a2ab-bfe072b7eff1"
Jul 28 20:47:43.326251 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:43.325945    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jtxnw" podUID="6c1be7ce-d683-4505-b781-5f739d0d82b5"
Jul 28 20:47:44.325972 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:44.325930    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:44.326412 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:44.326064    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8rwp" podUID="f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22"
Jul 28 20:47:45.327091 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:45.327051    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:45.327538 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:45.327208    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jtxnw" podUID="6c1be7ce-d683-4505-b781-5f739d0d82b5"
Jul 28 20:47:45.327538 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:45.327246    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:45.327538 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:45.327322    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qkd55" podUID="ba39afab-1128-4d15-a2ab-bfe072b7eff1"
Jul 28 20:47:46.325966 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:46.325927    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:46.326161 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:46.326061    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8rwp" podUID="f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22"
Jul 28 20:47:47.325909 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.325578    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:47.326500 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.325631    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:47.326500 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:47.326018    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jtxnw" podUID="6c1be7ce-d683-4505-b781-5f739d0d82b5"
Jul 28 20:47:47.326500 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:47.326077    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qkd55" podUID="ba39afab-1128-4d15-a2ab-bfe072b7eff1"
Jul 28 20:47:47.441590 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.441550    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g99mf" event={"ID":"a967c16a-fcff-43ad-b101-1b5835014439","Type":"ContainerStarted","Data":"521a6030217e482c46d3072ea74b484ec6f1e0000f386b9ec5546835830d6fcf"}
Jul 28 20:47:47.441774 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.441687    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g99mf" event={"ID":"a967c16a-fcff-43ad-b101-1b5835014439","Type":"ContainerStarted","Data":"a6922bbbe5c1eef0b4f0d012dbe946710ac0d99446b060727ba92cb35c515d8e"}
Jul 28 20:47:47.441774 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.441741    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g99mf" event={"ID":"a967c16a-fcff-43ad-b101-1b5835014439","Type":"ContainerStarted","Data":"00f185ad3c5da4d27bb01190db200c0c1c305b531c137bb153bdea8c93924340"}
Jul 28 20:47:47.441890 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.441788    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g99mf" event={"ID":"a967c16a-fcff-43ad-b101-1b5835014439","Type":"ContainerStarted","Data":"8535af2be311de9486ddfb75fd7c034488223ff193c4c511d6eb581c85ad157d"}
Jul 28 20:47:47.441890 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.441833    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g99mf" event={"ID":"a967c16a-fcff-43ad-b101-1b5835014439","Type":"ContainerStarted","Data":"9360c2bfe45162d4bf64ef982d780c643cbbc58e766377dd36c5b11ff66bb3d8"}
Jul 28 20:47:47.441890 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.441879    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g99mf" event={"ID":"a967c16a-fcff-43ad-b101-1b5835014439","Type":"ContainerStarted","Data":"ecbc7b3d5b4de8af5b50b3cc21edb53c3cbd98f926999e525835ffebfea55bca"}
Jul 28 20:47:47.443745 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.443689    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mv4mw" event={"ID":"65544b7e-687b-4ba8-afb4-ff571b8f4939","Type":"ContainerStarted","Data":"64c65b365e4243e003d55466ddf9388df45e8de426ca3f4ac5d1d726fb71f7c6"}
Jul 28 20:47:47.446170 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.446131    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5ngbd" event={"ID":"c20cff94-12ac-4cfe-861f-cb7557a7a0e6","Type":"ContainerStarted","Data":"60e024490a9bd973deb74dfafeb3bec24ffe22201f36297ebaacf7c583667db0"}
Jul 28 20:47:47.447836 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.447808    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822" event={"ID":"7b99cc53-e3f9-4970-a93a-1a052da4dd9d","Type":"ContainerStarted","Data":"4824c26e50752813fbb0dae6c48687952d87fa2ce1390173222477323f56c050"}
Jul 28 20:47:47.449647 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.449579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dkngp" event={"ID":"50035216-1c15-4307-ba45-212599d1e5ed","Type":"ContainerStarted","Data":"fc5e8f306ce2544bcd2e9576a02a912c904559b8813c68f850da1072f4103fd3"}
Jul 28 20:47:47.451257 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.451231    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wmxnd" event={"ID":"56d94607-9b1c-45ee-958e-9b66f2d946ea","Type":"ContainerStarted","Data":"84da85d1d0f01b3bec0a801226789b5481a3760b85c4514b77d5745c6800ce7b"}
Jul 28 20:47:47.453801 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.453318    2565 generic.go:356] "Generic (PLEG): container finished" podID="b81437ec-1218-4d56-9dc0-5f394a5e45f4" containerID="803c0aa3fcc66d2b0e4b7764dba2188763e30c9eb4ab8a4056f0ad2e058c4258" exitCode=0
Jul 28 20:47:47.453801 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.453522    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7vsq" event={"ID":"b81437ec-1218-4d56-9dc0-5f394a5e45f4","Type":"ContainerDied","Data":"803c0aa3fcc66d2b0e4b7764dba2188763e30c9eb4ab8a4056f0ad2e058c4258"}
Jul 28 20:47:47.455370 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.455334    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7prgk" event={"ID":"ff2108d7-6551-4eaa-8c51-a234010bc4ba","Type":"ContainerStarted","Data":"f174a171d758206a68a2355ef19aa19932cdce2701933e78cf666bf49a2764ad"}
Jul 28 20:47:47.470380 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.470324    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-170.ec2.internal" podStartSLOduration=21.470304721 podStartE2EDuration="21.470304721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:30.421435453 +0000 UTC m=+5.645778442" watchObservedRunningTime="2026-07-28 20:47:47.470304721 +0000 UTC m=+22.694647709"
Jul 28 20:47:47.470550 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.470422    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mv4mw" podStartSLOduration=3.875204133 podStartE2EDuration="22.470416649s" totalImagesPullingTime="18.595212516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:47.469762024 +0000 UTC m=+22.694105010" watchObservedRunningTime="2026-07-28 20:47:47.470416649 +0000 UTC m=+22.694759638"
Jul 28 20:47:47.487755 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.487709    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dkngp" podStartSLOduration=11.801599898 podStartE2EDuration="22.487690411s" totalImagesPullingTime="10.686090513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:47.487326931 +0000 UTC m=+22.711669917" watchObservedRunningTime="2026-07-28 20:47:47.487690411 +0000 UTC m=+22.712033397"
Jul 28 20:47:47.524567 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.524466    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5ngbd" podStartSLOduration=3.997183009 podStartE2EDuration="22.524444417s" totalImagesPullingTime="18.527261408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:47.503440073 +0000 UTC m=+22.727783059" watchObservedRunningTime="2026-07-28 20:47:47.524444417 +0000 UTC m=+22.748787404"
Jul 28 20:47:47.525062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.525029    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wmxnd" podStartSLOduration=3.98411367 podStartE2EDuration="22.525016873s" totalImagesPullingTime="18.540903203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:47.524415471 +0000 UTC m=+22.748758456" watchObservedRunningTime="2026-07-28 20:47:47.525016873 +0000 UTC m=+22.749359858"
Jul 28 20:47:47.577122 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:47.577075    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7prgk" podStartSLOduration=4.037577709 podStartE2EDuration="22.577059s" totalImagesPullingTime="18.539481291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:47.576876231 +0000 UTC m=+22.801219218" watchObservedRunningTime="2026-07-28 20:47:47.577059 +0000 UTC m=+22.801401986"
Jul 28 20:47:48.203796 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:48.203773    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 20:47:48.259309 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:48.259173    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T20:47:48.203790747Z","UUID":"d03a9802-75b5-4914-b636-66c8b193ed2f","Handler":null,"Name":"","Endpoint":""}
Jul 28 20:47:48.261339 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:48.261106    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 20:47:48.261339 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:48.261147    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 20:47:48.325839 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:48.325758    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:48.326020 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:48.325890    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8rwp" podUID="f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22"
Jul 28 20:47:48.460654 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:48.460609    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gkhq2" event={"ID":"b35a69ee-6fa5-4373-aadc-51e9aa800ff0","Type":"ContainerStarted","Data":"75d2779efb2940f3db8e6f46c9e8e646d72e5aa914515c11ae2af55a77401732"}
Jul 28 20:47:48.462799 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:48.462767    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822" event={"ID":"7b99cc53-e3f9-4970-a93a-1a052da4dd9d","Type":"ContainerStarted","Data":"f4f87f5d1ff003bc725a4927df7acf8469e07e46250ff32eeb654fb115e40370"}
Jul 28 20:47:49.325784 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:49.325748    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:49.326160 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:49.325748    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:49.326160 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:49.325888    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qkd55" podUID="ba39afab-1128-4d15-a2ab-bfe072b7eff1"
Jul 28 20:47:49.326160 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:49.325979    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jtxnw" podUID="6c1be7ce-d683-4505-b781-5f739d0d82b5"
Jul 28 20:47:49.467819 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:49.467777    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822" event={"ID":"7b99cc53-e3f9-4970-a93a-1a052da4dd9d","Type":"ContainerStarted","Data":"45f46310459bc5fe0001ba114f3a67f695a4ac52c7134497aba4b786a5707f6f"}
Jul 28 20:47:49.490071 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:49.490011    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gkhq2" podStartSLOduration=5.964847696 podStartE2EDuration="24.489989627s" totalImagesPullingTime="18.525141931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:48.479031665 +0000 UTC m=+23.703374651" watchObservedRunningTime="2026-07-28 20:47:49.489989627 +0000 UTC m=+24.714332615"
Jul 28 20:47:49.490353 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:49.490321    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xs822" podStartSLOduration=3.662869459 podStartE2EDuration="24.490313087s" totalImagesPullingTime="20.827443628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:49.489619927 +0000 UTC m=+24.713962914" watchObservedRunningTime="2026-07-28 20:47:49.490313087 +0000 UTC m=+24.714656074"
Jul 28 20:47:50.325592 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:50.325564    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:50.325774 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:50.325701    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8rwp" podUID="f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22"
Jul 28 20:47:50.473947 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:50.473907    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g99mf" event={"ID":"a967c16a-fcff-43ad-b101-1b5835014439","Type":"ContainerStarted","Data":"badc11136547b6e2952b8bab7a2959a64e71c5fafb14163cd0f6d91c6fb11f7f"}
Jul 28 20:47:51.109228 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:51.108997    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dkngp"
Jul 28 20:47:51.109778 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:51.109755    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dkngp"
Jul 28 20:47:51.325469 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:51.325438    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:51.325469 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:51.325465    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:51.325712 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:51.325550    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qkd55" podUID="ba39afab-1128-4d15-a2ab-bfe072b7eff1"
Jul 28 20:47:51.325712 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:51.325668    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jtxnw" podUID="6c1be7ce-d683-4505-b781-5f739d0d82b5"
Jul 28 20:47:51.475751 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:51.475671    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dkngp"
Jul 28 20:47:51.476285 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:51.476265    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dkngp"
Jul 28 20:47:52.325941 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:52.325691    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:52.326090 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:52.325969    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8rwp" podUID="f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22"
Jul 28 20:47:52.480050 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:52.480017    2565 generic.go:356] "Generic (PLEG): container finished" podID="b81437ec-1218-4d56-9dc0-5f394a5e45f4" containerID="5dba4f1f78300a69e03688e599f83899243b2751af1bbdcc59897cab2772fbc6" exitCode=0
Jul 28 20:47:52.480917 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:52.480109    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7vsq" event={"ID":"b81437ec-1218-4d56-9dc0-5f394a5e45f4","Type":"ContainerDied","Data":"5dba4f1f78300a69e03688e599f83899243b2751af1bbdcc59897cab2772fbc6"}
Jul 28 20:47:52.484122 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:52.484091    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g99mf" event={"ID":"a967c16a-fcff-43ad-b101-1b5835014439","Type":"ContainerStarted","Data":"7af218fedc5f340c9c0c05d0f09c39a129fae457b32a1bb7a8b833cb23cafb62"}
Jul 28 20:47:52.543195 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:52.543145    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g99mf" podStartSLOduration=8.945897613 podStartE2EDuration="27.543128296s" totalImagesPullingTime="18.597230683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:47:52.54258927 +0000 UTC m=+27.766932249" watchObservedRunningTime="2026-07-28 20:47:52.543128296 +0000 UTC m=+27.767471281"
Jul 28 20:47:53.329171 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:53.329143    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:53.329472 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:53.329225    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:53.329472 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:53.329242    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qkd55" podUID="ba39afab-1128-4d15-a2ab-bfe072b7eff1"
Jul 28 20:47:53.329472 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:53.329371    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jtxnw" podUID="6c1be7ce-d683-4505-b781-5f739d0d82b5"
Jul 28 20:47:53.487088 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:53.487054    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:53.487548 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:53.487135    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:53.487548 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:53.487177    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:53.502031 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:53.502006    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:53.502175 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:53.502130    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:47:53.861513 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:53.861430    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d8rwp"]
Jul 28 20:47:53.861812 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:53.861595    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:53.861812 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:53.861730    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8rwp" podUID="f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22"
Jul 28 20:47:53.862233 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:53.862207    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jtxnw"]
Jul 28 20:47:53.862349 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:53.862334    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:53.862451 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:53.862429    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jtxnw" podUID="6c1be7ce-d683-4505-b781-5f739d0d82b5"
Jul 28 20:47:53.863473 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:53.863445    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qkd55"]
Jul 28 20:47:53.863587 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:53.863536    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:53.863661 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:53.863600    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qkd55" podUID="ba39afab-1128-4d15-a2ab-bfe072b7eff1"
Jul 28 20:47:54.490431 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:54.490395    2565 generic.go:356] "Generic (PLEG): container finished" podID="b81437ec-1218-4d56-9dc0-5f394a5e45f4" containerID="1a641e4e19f421f6a461258e4cbb5069c2c6ef71b4afae8cb7d20ddaa093b2c0" exitCode=0
Jul 28 20:47:54.490866 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:54.490472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7vsq" event={"ID":"b81437ec-1218-4d56-9dc0-5f394a5e45f4","Type":"ContainerDied","Data":"1a641e4e19f421f6a461258e4cbb5069c2c6ef71b4afae8cb7d20ddaa093b2c0"}
Jul 28 20:47:55.327376 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:55.327109    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:55.327567 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:55.327175    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:55.327567 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:55.327473    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8rwp" podUID="f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22"
Jul 28 20:47:55.327695 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:55.327569    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jtxnw" podUID="6c1be7ce-d683-4505-b781-5f739d0d82b5"
Jul 28 20:47:55.327695 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:55.327190    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:55.327695 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:55.327667    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qkd55" podUID="ba39afab-1128-4d15-a2ab-bfe072b7eff1"
Jul 28 20:47:55.495755 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:55.495722    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7vsq" event={"ID":"b81437ec-1218-4d56-9dc0-5f394a5e45f4","Type":"ContainerStarted","Data":"16131dc82c930727893612764e75b6c222f135ca21772dd6d00562608bffdceb"}
Jul 28 20:47:56.501773 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:56.501734    2565 generic.go:356] "Generic (PLEG): container finished" podID="b81437ec-1218-4d56-9dc0-5f394a5e45f4" containerID="16131dc82c930727893612764e75b6c222f135ca21772dd6d00562608bffdceb" exitCode=0
Jul 28 20:47:56.502323 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:56.501780    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7vsq" event={"ID":"b81437ec-1218-4d56-9dc0-5f394a5e45f4","Type":"ContainerDied","Data":"16131dc82c930727893612764e75b6c222f135ca21772dd6d00562608bffdceb"}
Jul 28 20:47:57.325474 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:57.325442    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:57.325699 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:57.325590    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:57.325699 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:57.325626    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:57.325824 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:57.325720    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8rwp" podUID="f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22"
Jul 28 20:47:57.325824 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:57.325593    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jtxnw" podUID="6c1be7ce-d683-4505-b781-5f739d0d82b5"
Jul 28 20:47:57.325928 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:57.325818    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qkd55" podUID="ba39afab-1128-4d15-a2ab-bfe072b7eff1"
Jul 28 20:47:58.961042 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:58.961008    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret\") pod \"global-pull-secret-syncer-d8rwp\" (UID: \"f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22\") " pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:58.961731 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:58.961180    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:58.961731 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:58.961256    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret podName:f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:30.961233759 +0000 UTC m=+66.185576759 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret") pod "global-pull-secret-syncer-d8rwp" (UID: "f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22") : object "kube-system"/"original-pull-secret" not registered
Jul 28 20:47:59.061986 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.061948    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnzhq\" (UniqueName: \"kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq\") pod \"network-check-target-qkd55\" (UID: \"ba39afab-1128-4d15-a2ab-bfe072b7eff1\") " pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:59.062179 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.062024    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs\") pod \"network-metrics-daemon-jtxnw\" (UID: \"6c1be7ce-d683-4505-b781-5f739d0d82b5\") " pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:59.062179 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:59.062152    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:59.062300 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:59.062220    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs podName:6c1be7ce-d683-4505-b781-5f739d0d82b5 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:31.062205374 +0000 UTC m=+66.286548342 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs") pod "network-metrics-daemon-jtxnw" (UID: "6c1be7ce-d683-4505-b781-5f739d0d82b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 20:47:59.062300 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:59.062151    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 20:47:59.062300 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:59.062271    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 20:47:59.062300 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:59.062286    2565 projected.go:196] Error preparing data for projected volume kube-api-access-fnzhq for pod openshift-network-diagnostics/network-check-target-qkd55: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:59.062505 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:59.062342    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq podName:ba39afab-1128-4d15-a2ab-bfe072b7eff1 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:31.062326733 +0000 UTC m=+66.286669715 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fnzhq" (UniqueName: "kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq") pod "network-check-target-qkd55" (UID: "ba39afab-1128-4d15-a2ab-bfe072b7eff1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 20:47:59.325267 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.325230    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:47:59.325429 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.325229    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:47:59.325429 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.325241    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:47:59.325429 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:59.325378    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qkd55" podUID="ba39afab-1128-4d15-a2ab-bfe072b7eff1"
Jul 28 20:47:59.325577 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:59.325446    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jtxnw" podUID="6c1be7ce-d683-4505-b781-5f739d0d82b5"
Jul 28 20:47:59.325577 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:59.325554    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d8rwp" podUID="f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22"
Jul 28 20:47:59.553622 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.553541    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-170.ec2.internal" event="NodeReady"
Jul 28 20:47:59.553794 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.553688    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 20:47:59.605072 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.605035    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x6qn9"]
Jul 28 20:47:59.610263 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.610238    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7g9v6"]
Jul 28 20:47:59.610404 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.610380    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:47:59.613229 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.613206    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 20:47:59.614153 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.614098    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7g9v6"
Jul 28 20:47:59.614271 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.614189    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 20:47:59.614556 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.614539    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ghnpw\""
Jul 28 20:47:59.616754 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.616732    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 20:47:59.616869 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.616755    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 20:47:59.619189 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.617731    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-m4gqb\""
Jul 28 20:47:59.619189 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.618212    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 20:47:59.619647 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.619336    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x6qn9"]
Jul 28 20:47:59.621825 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.621797    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7g9v6"]
Jul 28 20:47:59.768115 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.768074    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls\") pod \"dns-default-x6qn9\" (UID: \"41a72d4d-a128-4ed4-934f-0aebbe9825b6\") " pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:47:59.768115 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.768123    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4trc4\" (UniqueName: \"kubernetes.io/projected/41a72d4d-a128-4ed4-934f-0aebbe9825b6-kube-api-access-4trc4\") pod \"dns-default-x6qn9\" (UID: \"41a72d4d-a128-4ed4-934f-0aebbe9825b6\") " pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:47:59.768391 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.768173    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4692cbb4-e2af-4b0d-ab5e-ff1be75e2996-cert\") pod \"ingress-canary-7g9v6\" (UID: \"4692cbb4-e2af-4b0d-ab5e-ff1be75e2996\") " pod="openshift-ingress-canary/ingress-canary-7g9v6"
Jul 28 20:47:59.768391 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.768201    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41a72d4d-a128-4ed4-934f-0aebbe9825b6-tmp-dir\") pod \"dns-default-x6qn9\" (UID: \"41a72d4d-a128-4ed4-934f-0aebbe9825b6\") " pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:47:59.768391 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.768322    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scm7k\" (UniqueName: \"kubernetes.io/projected/4692cbb4-e2af-4b0d-ab5e-ff1be75e2996-kube-api-access-scm7k\") pod \"ingress-canary-7g9v6\" (UID: \"4692cbb4-e2af-4b0d-ab5e-ff1be75e2996\") " pod="openshift-ingress-canary/ingress-canary-7g9v6"
Jul 28 20:47:59.768391 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.768383    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/41a72d4d-a128-4ed4-934f-0aebbe9825b6-config-volume\") pod \"dns-default-x6qn9\" (UID: \"41a72d4d-a128-4ed4-934f-0aebbe9825b6\") " pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:47:59.869477 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.869374    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scm7k\" (UniqueName: \"kubernetes.io/projected/4692cbb4-e2af-4b0d-ab5e-ff1be75e2996-kube-api-access-scm7k\") pod \"ingress-canary-7g9v6\" (UID: \"4692cbb4-e2af-4b0d-ab5e-ff1be75e2996\") " pod="openshift-ingress-canary/ingress-canary-7g9v6"
Jul 28 20:47:59.869477 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.869443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/41a72d4d-a128-4ed4-934f-0aebbe9825b6-config-volume\") pod \"dns-default-x6qn9\" (UID: \"41a72d4d-a128-4ed4-934f-0aebbe9825b6\") " pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:47:59.869733 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.869485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls\") pod \"dns-default-x6qn9\" (UID: \"41a72d4d-a128-4ed4-934f-0aebbe9825b6\") " pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:47:59.869733 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:59.869574    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:47:59.869733 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.869584    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4trc4\" (UniqueName: \"kubernetes.io/projected/41a72d4d-a128-4ed4-934f-0aebbe9825b6-kube-api-access-4trc4\") pod \"dns-default-x6qn9\" (UID: \"41a72d4d-a128-4ed4-934f-0aebbe9825b6\") " pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:47:59.869733 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:47:59.869672    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls podName:41a72d4d-a128-4ed4-934f-0aebbe9825b6 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:00.369630263 +0000 UTC m=+35.593973229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls") pod "dns-default-x6qn9" (UID: "41a72d4d-a128-4ed4-934f-0aebbe9825b6") : secret "dns-default-metrics-tls" not found
Jul 28 20:47:59.869733 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.869693    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4692cbb4-e2af-4b0d-ab5e-ff1be75e2996-cert\") pod \"ingress-canary-7g9v6\" (UID: \"4692cbb4-e2af-4b0d-ab5e-ff1be75e2996\") " pod="openshift-ingress-canary/ingress-canary-7g9v6"
Jul 28 20:47:59.869733 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.869721    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41a72d4d-a128-4ed4-934f-0aebbe9825b6-tmp-dir\") pod \"dns-default-x6qn9\" (UID: \"41a72d4d-a128-4ed4-934f-0aebbe9825b6\") " pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:47:59.870076 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.870055    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41a72d4d-a128-4ed4-934f-0aebbe9825b6-tmp-dir\") pod \"dns-default-x6qn9\" (UID: \"41a72d4d-a128-4ed4-934f-0aebbe9825b6\") " pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:47:59.870159 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.870132    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/41a72d4d-a128-4ed4-934f-0aebbe9825b6-config-volume\") pod \"dns-default-x6qn9\" (UID: \"41a72d4d-a128-4ed4-934f-0aebbe9825b6\") " pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:47:59.874737 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.874713    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4692cbb4-e2af-4b0d-ab5e-ff1be75e2996-cert\") pod \"ingress-canary-7g9v6\" (UID: \"4692cbb4-e2af-4b0d-ab5e-ff1be75e2996\") " pod="openshift-ingress-canary/ingress-canary-7g9v6"
Jul 28 20:47:59.880497 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.880470    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scm7k\" (UniqueName: \"kubernetes.io/projected/4692cbb4-e2af-4b0d-ab5e-ff1be75e2996-kube-api-access-scm7k\") pod \"ingress-canary-7g9v6\" (UID: \"4692cbb4-e2af-4b0d-ab5e-ff1be75e2996\") " pod="openshift-ingress-canary/ingress-canary-7g9v6"
Jul 28 20:47:59.880661 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.880624    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4trc4\" (UniqueName: \"kubernetes.io/projected/41a72d4d-a128-4ed4-934f-0aebbe9825b6-kube-api-access-4trc4\") pod \"dns-default-x6qn9\" (UID: \"41a72d4d-a128-4ed4-934f-0aebbe9825b6\") " pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:47:59.928805 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:47:59.928767    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7g9v6"
Jul 28 20:48:00.375375 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:00.375337    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls\") pod \"dns-default-x6qn9\" (UID: \"41a72d4d-a128-4ed4-934f-0aebbe9825b6\") " pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:48:00.376021 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:00.375510    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:48:00.376021 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:00.375589    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls podName:41a72d4d-a128-4ed4-934f-0aebbe9825b6 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:01.375569717 +0000 UTC m=+36.599912681 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls") pod "dns-default-x6qn9" (UID: "41a72d4d-a128-4ed4-934f-0aebbe9825b6") : secret "dns-default-metrics-tls" not found
Jul 28 20:48:00.567950 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:00.567716    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7g9v6"]
Jul 28 20:48:00.571244 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:48:00.571215    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4692cbb4_e2af_4b0d_ab5e_ff1be75e2996.slice/crio-ad7a8e4733c8243b5312a2a6ace240601f6d87718ec8faa39ecd94e947b88e0c WatchSource:0}: Error finding container ad7a8e4733c8243b5312a2a6ace240601f6d87718ec8faa39ecd94e947b88e0c: Status 404 returned error can't find the container with id ad7a8e4733c8243b5312a2a6ace240601f6d87718ec8faa39ecd94e947b88e0c
Jul 28 20:48:01.325570 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:01.325539    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:48:01.325768 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:01.325538    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:48:01.325768 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:01.325539    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:48:01.329996 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:01.329972    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 20:48:01.330142 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:01.330072    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 20:48:01.330142 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:01.330069    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 20:48:01.330546 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:01.330381    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j52pw\""
Jul 28 20:48:01.330546 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:01.330474    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 20:48:01.331100 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:01.331081    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gvjsj\""
Jul 28 20:48:01.383793 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:01.383752    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls\") pod \"dns-default-x6qn9\" (UID: \"41a72d4d-a128-4ed4-934f-0aebbe9825b6\") " pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:48:01.384273 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:01.383893    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:48:01.384273 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:01.383948    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls podName:41a72d4d-a128-4ed4-934f-0aebbe9825b6 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:03.383932695 +0000 UTC m=+38.608275658 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls") pod "dns-default-x6qn9" (UID: "41a72d4d-a128-4ed4-934f-0aebbe9825b6") : secret "dns-default-metrics-tls" not found
Jul 28 20:48:01.519268 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:01.519236    2565 generic.go:356] "Generic (PLEG): container finished" podID="b81437ec-1218-4d56-9dc0-5f394a5e45f4" containerID="5d2f69178be859e8b314de5d117eaacfd62a79cce7b624f35136b3b544b88ddb" exitCode=0
Jul 28 20:48:01.519458 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:01.519319    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7vsq" event={"ID":"b81437ec-1218-4d56-9dc0-5f394a5e45f4","Type":"ContainerDied","Data":"5d2f69178be859e8b314de5d117eaacfd62a79cce7b624f35136b3b544b88ddb"}
Jul 28 20:48:01.520851 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:01.520651    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7g9v6" event={"ID":"4692cbb4-e2af-4b0d-ab5e-ff1be75e2996","Type":"ContainerStarted","Data":"ad7a8e4733c8243b5312a2a6ace240601f6d87718ec8faa39ecd94e947b88e0c"}
Jul 28 20:48:02.527460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:02.527418    2565 generic.go:356] "Generic (PLEG): container finished" podID="b81437ec-1218-4d56-9dc0-5f394a5e45f4" containerID="8ebb0c880f5a2f56942675a6e90cf350dbba63b31875629e0e2047ab14348c80" exitCode=0
Jul 28 20:48:02.527912 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:02.527506    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7vsq" event={"ID":"b81437ec-1218-4d56-9dc0-5f394a5e45f4","Type":"ContainerDied","Data":"8ebb0c880f5a2f56942675a6e90cf350dbba63b31875629e0e2047ab14348c80"}
Jul 28 20:48:03.402935 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:03.402516    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls\") pod \"dns-default-x6qn9\" (UID: \"41a72d4d-a128-4ed4-934f-0aebbe9825b6\") " pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:48:03.402935 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:03.402714    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:48:03.402935 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:03.402800    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls podName:41a72d4d-a128-4ed4-934f-0aebbe9825b6 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:07.402777414 +0000 UTC m=+42.627120402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls") pod "dns-default-x6qn9" (UID: "41a72d4d-a128-4ed4-934f-0aebbe9825b6") : secret "dns-default-metrics-tls" not found
Jul 28 20:48:03.532721 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:03.532684    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7vsq" event={"ID":"b81437ec-1218-4d56-9dc0-5f394a5e45f4","Type":"ContainerStarted","Data":"a9586530348d0bce033b4867fdad1e6caf2fa261d49d9d548f5478b63bab462b"}
Jul 28 20:48:03.536015 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:03.535985    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7g9v6" event={"ID":"4692cbb4-e2af-4b0d-ab5e-ff1be75e2996","Type":"ContainerStarted","Data":"2e0aa4e1ac70bb60ec0af6869358d418b3248ff05200e54439e4a41aaac2de4f"}
Jul 28 20:48:03.560330 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:03.560274    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k7vsq" podStartSLOduration=9.698545745 podStartE2EDuration="38.560255311s" totalImagesPullingTime="28.861709566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:48:03.55836751 +0000 UTC m=+38.782710496" watchObservedRunningTime="2026-07-28 20:48:03.560255311 +0000 UTC m=+38.784598298"
Jul 28 20:48:03.575481 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:03.575429    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7g9v6" podStartSLOduration=2.239652623 podStartE2EDuration="4.575413546s" totalImagesPullingTime="2.335760923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:48:03.57472069 +0000 UTC m=+38.799063675" watchObservedRunningTime="2026-07-28 20:48:03.575413546 +0000 UTC m=+38.799756531"
Jul 28 20:48:07.431549 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:07.431509    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls\") pod \"dns-default-x6qn9\" (UID: \"41a72d4d-a128-4ed4-934f-0aebbe9825b6\") " pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:48:07.432088 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:07.431687    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:48:07.432088 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:07.431751    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls podName:41a72d4d-a128-4ed4-934f-0aebbe9825b6 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:15.431733901 +0000 UTC m=+50.656076869 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls") pod "dns-default-x6qn9" (UID: "41a72d4d-a128-4ed4-934f-0aebbe9825b6") : secret "dns-default-metrics-tls" not found
Jul 28 20:48:15.486761 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:15.486715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls\") pod \"dns-default-x6qn9\" (UID: \"41a72d4d-a128-4ed4-934f-0aebbe9825b6\") " pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:48:15.487235 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:15.486866    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:48:15.487235 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:15.486930    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls podName:41a72d4d-a128-4ed4-934f-0aebbe9825b6 nodeName:}" failed. No retries permitted until 2026-07-28 20:48:31.486912799 +0000 UTC m=+66.711255763 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls") pod "dns-default-x6qn9" (UID: "41a72d4d-a128-4ed4-934f-0aebbe9825b6") : secret "dns-default-metrics-tls" not found
Jul 28 20:48:20.928586 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:20.928549    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd"]
Jul 28 20:48:20.930715 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:20.930697    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd"
Jul 28 20:48:20.934276 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:20.934239    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 28 20:48:20.934276 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:20.934272    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 20:48:20.934544 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:20.934285    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 20:48:20.934544 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:20.934309    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 28 20:48:20.935275 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:20.935256    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-sr5d7\""
Jul 28 20:48:20.942921 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:20.942895    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd"]
Jul 28 20:48:21.023906 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.023863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-rvtwd\" (UID: \"18a1dfd7-9d6b-4bb8-a22c-956fca90189d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd"
Jul 28 20:48:21.023906 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.023913    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-rvtwd\" (UID: \"18a1dfd7-9d6b-4bb8-a22c-956fca90189d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd"
Jul 28 20:48:21.024139 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.023932    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drs6z\" (UniqueName: \"kubernetes.io/projected/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-kube-api-access-drs6z\") pod \"cluster-monitoring-operator-548b489cb8-rvtwd\" (UID: \"18a1dfd7-9d6b-4bb8-a22c-956fca90189d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd"
Jul 28 20:48:21.077130 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.077095    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bb96c4ff7-hbx9q"]
Jul 28 20:48:21.079145 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.079130    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.083296 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.083275    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 20:48:21.083600 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.083586    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 20:48:21.083832 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.083815    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 20:48:21.083879 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.083830    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-52n7n\""
Jul 28 20:48:21.089914 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.089892    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 20:48:21.095460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.095438    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bb96c4ff7-hbx9q"]
Jul 28 20:48:21.124848 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.124812    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a635f930-2655-4762-b3f6-67e2f947dc8c-image-registry-private-configuration\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.124848 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.124848    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.125092 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.124872    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-certificates\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.125092 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.124910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a635f930-2655-4762-b3f6-67e2f947dc8c-ca-trust-extracted\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.125092 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.124932    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xk9d\" (UniqueName: \"kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-kube-api-access-5xk9d\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.125092 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.124962    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-bound-sa-token\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.125092 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.125057    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-rvtwd\" (UID: \"18a1dfd7-9d6b-4bb8-a22c-956fca90189d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd"
Jul 28 20:48:21.125336 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.125108    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a635f930-2655-4762-b3f6-67e2f947dc8c-trusted-ca\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.125336 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.125152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-rvtwd\" (UID: \"18a1dfd7-9d6b-4bb8-a22c-956fca90189d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd"
Jul 28 20:48:21.125336 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:21.125182    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 20:48:21.125336 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.125179    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drs6z\" (UniqueName: \"kubernetes.io/projected/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-kube-api-access-drs6z\") pod \"cluster-monitoring-operator-548b489cb8-rvtwd\" (UID: \"18a1dfd7-9d6b-4bb8-a22c-956fca90189d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd"
Jul 28 20:48:21.125336 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:21.125271    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls podName:18a1dfd7-9d6b-4bb8-a22c-956fca90189d nodeName:}" failed. No retries permitted until 2026-07-28 20:48:21.625228715 +0000 UTC m=+56.849571680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-rvtwd" (UID: "18a1dfd7-9d6b-4bb8-a22c-956fca90189d") : secret "cluster-monitoring-operator-tls" not found
Jul 28 20:48:21.125336 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.125294    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a635f930-2655-4762-b3f6-67e2f947dc8c-installation-pull-secrets\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.125821 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.125803    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-rvtwd\" (UID: \"18a1dfd7-9d6b-4bb8-a22c-956fca90189d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd"
Jul 28 20:48:21.135675 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.135654    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drs6z\" (UniqueName: \"kubernetes.io/projected/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-kube-api-access-drs6z\") pod \"cluster-monitoring-operator-548b489cb8-rvtwd\" (UID: \"18a1dfd7-9d6b-4bb8-a22c-956fca90189d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd"
Jul 28 20:48:21.226100 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.226048    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a635f930-2655-4762-b3f6-67e2f947dc8c-image-registry-private-configuration\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.226100 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.226099    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.226384 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.226121    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-certificates\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.226384 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.226151    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a635f930-2655-4762-b3f6-67e2f947dc8c-ca-trust-extracted\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.226384 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.226168    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xk9d\" (UniqueName: \"kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-kube-api-access-5xk9d\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.226384 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.226193    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-bound-sa-token\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.226384 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:21.226218    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 20:48:21.226384 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:21.226242    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bb96c4ff7-hbx9q: secret "image-registry-tls" not found
Jul 28 20:48:21.226384 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:21.226328    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls podName:a635f930-2655-4762-b3f6-67e2f947dc8c nodeName:}" failed. No retries permitted until 2026-07-28 20:48:21.726311557 +0000 UTC m=+56.950654521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls") pod "image-registry-bb96c4ff7-hbx9q" (UID: "a635f930-2655-4762-b3f6-67e2f947dc8c") : secret "image-registry-tls" not found
Jul 28 20:48:21.226711 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.226392    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a635f930-2655-4762-b3f6-67e2f947dc8c-trusted-ca\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.226711 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.226445    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a635f930-2655-4762-b3f6-67e2f947dc8c-installation-pull-secrets\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.226711 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.226611    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a635f930-2655-4762-b3f6-67e2f947dc8c-ca-trust-extracted\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.226976 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.226933    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-certificates\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.227260 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.227240    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a635f930-2655-4762-b3f6-67e2f947dc8c-trusted-ca\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.228615 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.228594    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a635f930-2655-4762-b3f6-67e2f947dc8c-image-registry-private-configuration\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.228743 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.228685    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a635f930-2655-4762-b3f6-67e2f947dc8c-installation-pull-secrets\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.235416 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.235391    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-bound-sa-token\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.236134 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.236114    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xk9d\" (UniqueName: \"kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-kube-api-access-5xk9d\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.629034 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.628939    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-rvtwd\" (UID: \"18a1dfd7-9d6b-4bb8-a22c-956fca90189d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd"
Jul 28 20:48:21.629182 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:21.629092    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 20:48:21.629182 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:21.629162    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls podName:18a1dfd7-9d6b-4bb8-a22c-956fca90189d nodeName:}" failed. No retries permitted until 2026-07-28 20:48:22.629144215 +0000 UTC m=+57.853487191 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-rvtwd" (UID: "18a1dfd7-9d6b-4bb8-a22c-956fca90189d") : secret "cluster-monitoring-operator-tls" not found
Jul 28 20:48:21.729790 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:21.729755    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:21.729930 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:21.729910    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 20:48:21.729969 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:21.729933    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bb96c4ff7-hbx9q: secret "image-registry-tls" not found
Jul 28 20:48:21.730011 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:21.730003    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls podName:a635f930-2655-4762-b3f6-67e2f947dc8c nodeName:}" failed. No retries permitted until 2026-07-28 20:48:22.729986906 +0000 UTC m=+57.954329870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls") pod "image-registry-bb96c4ff7-hbx9q" (UID: "a635f930-2655-4762-b3f6-67e2f947dc8c") : secret "image-registry-tls" not found
Jul 28 20:48:22.637295 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:22.637252    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-rvtwd\" (UID: \"18a1dfd7-9d6b-4bb8-a22c-956fca90189d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd"
Jul 28 20:48:22.637708 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:22.637406    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 20:48:22.637708 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:22.637484    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls podName:18a1dfd7-9d6b-4bb8-a22c-956fca90189d nodeName:}" failed. No retries permitted until 2026-07-28 20:48:24.637467678 +0000 UTC m=+59.861810641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-rvtwd" (UID: "18a1dfd7-9d6b-4bb8-a22c-956fca90189d") : secret "cluster-monitoring-operator-tls" not found
Jul 28 20:48:22.738400 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:22.738354    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:22.738574 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:22.738485    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 20:48:22.738574 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:22.738498    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bb96c4ff7-hbx9q: secret "image-registry-tls" not found
Jul 28 20:48:22.738574 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:22.738557    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls podName:a635f930-2655-4762-b3f6-67e2f947dc8c nodeName:}" failed. No retries permitted until 2026-07-28 20:48:24.738542591 +0000 UTC m=+59.962885555 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls") pod "image-registry-bb96c4ff7-hbx9q" (UID: "a635f930-2655-4762-b3f6-67e2f947dc8c") : secret "image-registry-tls" not found
Jul 28 20:48:24.652090 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:24.652047    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-rvtwd\" (UID: \"18a1dfd7-9d6b-4bb8-a22c-956fca90189d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd"
Jul 28 20:48:24.652494 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:24.652213    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 20:48:24.652494 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:24.652287    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls podName:18a1dfd7-9d6b-4bb8-a22c-956fca90189d nodeName:}" failed. No retries permitted until 2026-07-28 20:48:28.652267143 +0000 UTC m=+63.876610126 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-rvtwd" (UID: "18a1dfd7-9d6b-4bb8-a22c-956fca90189d") : secret "cluster-monitoring-operator-tls" not found
Jul 28 20:48:24.752848 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:24.752808    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:24.753004 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:24.752962    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 20:48:24.753004 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:24.752977    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bb96c4ff7-hbx9q: secret "image-registry-tls" not found
Jul 28 20:48:24.753075 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:24.753036    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls podName:a635f930-2655-4762-b3f6-67e2f947dc8c nodeName:}" failed. No retries permitted until 2026-07-28 20:48:28.753018194 +0000 UTC m=+63.977361169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls") pod "image-registry-bb96c4ff7-hbx9q" (UID: "a635f930-2655-4762-b3f6-67e2f947dc8c") : secret "image-registry-tls" not found
Jul 28 20:48:25.512583 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.512551    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g99mf"
Jul 28 20:48:25.811840 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.811766    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7vtw8"]
Jul 28 20:48:25.835799 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.835768    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7vtw8"]
Jul 28 20:48:25.835985 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.835892    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:25.838153 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.838134    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9jg6j\""
Jul 28 20:48:25.838365 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.838352    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 20:48:25.838936 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.838918    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 20:48:25.838983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.838922    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 20:48:25.841697 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.841676    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 20:48:25.860782 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.860739    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/100dd861-fcc1-45da-ac5b-0797754c195e-data-volume\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:25.860935 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.860795    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:25.860935 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.860841    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/100dd861-fcc1-45da-ac5b-0797754c195e-crio-socket\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:25.860935 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.860883    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4nc9\" (UniqueName: \"kubernetes.io/projected/100dd861-fcc1-45da-ac5b-0797754c195e-kube-api-access-w4nc9\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:25.861099 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.860934    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/100dd861-fcc1-45da-ac5b-0797754c195e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:25.961962 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.961910    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/100dd861-fcc1-45da-ac5b-0797754c195e-data-volume\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:25.961962 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.961968    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:25.962218 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:25.962061    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:25.962218 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.962109    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/100dd861-fcc1-45da-ac5b-0797754c195e-crio-socket\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:25.962218 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:25.962120    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls podName:100dd861-fcc1-45da-ac5b-0797754c195e nodeName:}" failed. No retries permitted until 2026-07-28 20:48:26.462104598 +0000 UTC m=+61.686447562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7vtw8" (UID: "100dd861-fcc1-45da-ac5b-0797754c195e") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:25.962218 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.962158    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4nc9\" (UniqueName: \"kubernetes.io/projected/100dd861-fcc1-45da-ac5b-0797754c195e-kube-api-access-w4nc9\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:25.962218 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.962183    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/100dd861-fcc1-45da-ac5b-0797754c195e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:25.962218 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.962205    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/100dd861-fcc1-45da-ac5b-0797754c195e-crio-socket\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:25.962423 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:25.962288    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 20:48:25.962423 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.962290    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/100dd861-fcc1-45da-ac5b-0797754c195e-data-volume\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:25.962423 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:25.962348    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/100dd861-fcc1-45da-ac5b-0797754c195e-kube-rbac-proxy-cm podName:100dd861-fcc1-45da-ac5b-0797754c195e nodeName:}" failed. No retries permitted until 2026-07-28 20:48:26.462334902 +0000 UTC m=+61.686677866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/100dd861-fcc1-45da-ac5b-0797754c195e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7vtw8" (UID: "100dd861-fcc1-45da-ac5b-0797754c195e") : configmap "kube-rbac-proxy" not found
Jul 28 20:48:25.977652 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:25.977613    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4nc9\" (UniqueName: \"kubernetes.io/projected/100dd861-fcc1-45da-ac5b-0797754c195e-kube-api-access-w4nc9\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:26.466850 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:26.466813    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/100dd861-fcc1-45da-ac5b-0797754c195e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:26.467062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:26.466931    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:26.467062 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:26.466938    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 20:48:26.467062 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:26.467016    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/100dd861-fcc1-45da-ac5b-0797754c195e-kube-rbac-proxy-cm podName:100dd861-fcc1-45da-ac5b-0797754c195e nodeName:}" failed. No retries permitted until 2026-07-28 20:48:27.466999108 +0000 UTC m=+62.691342093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/100dd861-fcc1-45da-ac5b-0797754c195e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7vtw8" (UID: "100dd861-fcc1-45da-ac5b-0797754c195e") : configmap "kube-rbac-proxy" not found
Jul 28 20:48:26.467171 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:26.467085    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:26.467171 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:26.467139    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls podName:100dd861-fcc1-45da-ac5b-0797754c195e nodeName:}" failed. No retries permitted until 2026-07-28 20:48:27.467126508 +0000 UTC m=+62.691469472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7vtw8" (UID: "100dd861-fcc1-45da-ac5b-0797754c195e") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:26.920616 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:26.920539    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-cgkpx"]
Jul 28 20:48:26.922953 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:26.922935    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-cgkpx"
Jul 28 20:48:26.934758 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:26.934731    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 20:48:26.934855 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:26.934741    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 20:48:26.934892 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:26.934886    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7xbv4\""
Jul 28 20:48:26.936348 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:26.936326    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-cgkpx"]
Jul 28 20:48:26.970625 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:26.970584    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9nxx\" (UniqueName: \"kubernetes.io/projected/4c5e2247-0906-44a8-b074-f9f03891fff5-kube-api-access-n9nxx\") pod \"migrator-6865c9967-cgkpx\" (UID: \"4c5e2247-0906-44a8-b074-f9f03891fff5\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-cgkpx"
Jul 28 20:48:27.071110 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:27.071073    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nxx\" (UniqueName: \"kubernetes.io/projected/4c5e2247-0906-44a8-b074-f9f03891fff5-kube-api-access-n9nxx\") pod \"migrator-6865c9967-cgkpx\" (UID: \"4c5e2247-0906-44a8-b074-f9f03891fff5\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-cgkpx"
Jul 28 20:48:27.079741 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:27.079711    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9nxx\" (UniqueName: \"kubernetes.io/projected/4c5e2247-0906-44a8-b074-f9f03891fff5-kube-api-access-n9nxx\") pod \"migrator-6865c9967-cgkpx\" (UID: \"4c5e2247-0906-44a8-b074-f9f03891fff5\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-cgkpx"
Jul 28 20:48:27.231836 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:27.231792    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-cgkpx"
Jul 28 20:48:27.371000 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:27.370968    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-cgkpx"]
Jul 28 20:48:27.375033 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:48:27.375003    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5e2247_0906_44a8_b074_f9f03891fff5.slice/crio-230463b9045312a6eb68631a24519927dcdcaa5a0ce13d0dae12594f9095cd0b WatchSource:0}: Error finding container 230463b9045312a6eb68631a24519927dcdcaa5a0ce13d0dae12594f9095cd0b: Status 404 returned error can't find the container with id 230463b9045312a6eb68631a24519927dcdcaa5a0ce13d0dae12594f9095cd0b
Jul 28 20:48:27.474225 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:27.474185    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/100dd861-fcc1-45da-ac5b-0797754c195e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:27.474421 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:27.474329    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 20:48:27.474421 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:27.474360    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:27.474421 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:27.474394    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/100dd861-fcc1-45da-ac5b-0797754c195e-kube-rbac-proxy-cm podName:100dd861-fcc1-45da-ac5b-0797754c195e nodeName:}" failed. No retries permitted until 2026-07-28 20:48:29.474377964 +0000 UTC m=+64.698720929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/100dd861-fcc1-45da-ac5b-0797754c195e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7vtw8" (UID: "100dd861-fcc1-45da-ac5b-0797754c195e") : configmap "kube-rbac-proxy" not found
Jul 28 20:48:27.474542 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:27.474457    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:27.474542 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:27.474515    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls podName:100dd861-fcc1-45da-ac5b-0797754c195e nodeName:}" failed. No retries permitted until 2026-07-28 20:48:29.474499341 +0000 UTC m=+64.698842324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7vtw8" (UID: "100dd861-fcc1-45da-ac5b-0797754c195e") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:27.593254 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:27.593164    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-cgkpx" event={"ID":"4c5e2247-0906-44a8-b074-f9f03891fff5","Type":"ContainerStarted","Data":"230463b9045312a6eb68631a24519927dcdcaa5a0ce13d0dae12594f9095cd0b"}
Jul 28 20:48:27.752393 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:27.752366    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7prgk_ff2108d7-6551-4eaa-8c51-a234010bc4ba/dns-node-resolver/0.log"
Jul 28 20:48:28.553228 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:28.553196    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5ngbd_c20cff94-12ac-4cfe-861f-cb7557a7a0e6/node-ca/0.log"
Jul 28 20:48:28.597564 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:28.597519    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-cgkpx" event={"ID":"4c5e2247-0906-44a8-b074-f9f03891fff5","Type":"ContainerStarted","Data":"e3f3aea8abfe4dd42466a196d089ad5d0eec68ec9534ab35c7276f00a34fc2ae"}
Jul 28 20:48:28.597753 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:28.597623    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-cgkpx" event={"ID":"4c5e2247-0906-44a8-b074-f9f03891fff5","Type":"ContainerStarted","Data":"a5d59914d755018390571bda39d976b57753360d121cb2d8db991bb5b9194647"}
Jul 28 20:48:28.684534 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:28.684506    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-rvtwd\" (UID: \"18a1dfd7-9d6b-4bb8-a22c-956fca90189d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd"
Jul 28 20:48:28.684698 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:28.684662    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 20:48:28.684747 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:28.684726    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls podName:18a1dfd7-9d6b-4bb8-a22c-956fca90189d nodeName:}" failed. No retries permitted until 2026-07-28 20:48:36.684711612 +0000 UTC m=+71.909054575 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-rvtwd" (UID: "18a1dfd7-9d6b-4bb8-a22c-956fca90189d") : secret "cluster-monitoring-operator-tls" not found
Jul 28 20:48:28.785884 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:28.785837    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:28.786054 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:28.786002    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 20:48:28.786054 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:28.786024    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bb96c4ff7-hbx9q: secret "image-registry-tls" not found
Jul 28 20:48:28.786123 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:28.786079    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls podName:a635f930-2655-4762-b3f6-67e2f947dc8c nodeName:}" failed. No retries permitted until 2026-07-28 20:48:36.786061786 +0000 UTC m=+72.010404750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls") pod "image-registry-bb96c4ff7-hbx9q" (UID: "a635f930-2655-4762-b3f6-67e2f947dc8c") : secret "image-registry-tls" not found
Jul 28 20:48:29.352786 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:29.352748    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7g9v6_4692cbb4-e2af-4b0d-ab5e-ff1be75e2996/serve-healthcheck-canary/0.log"
Jul 28 20:48:29.491847 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:29.491811    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:29.491968 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:29.491873    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/100dd861-fcc1-45da-ac5b-0797754c195e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:29.491968 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:29.491939    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 20:48:29.491968 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:29.491958    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:29.492062 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:29.491991    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/100dd861-fcc1-45da-ac5b-0797754c195e-kube-rbac-proxy-cm podName:100dd861-fcc1-45da-ac5b-0797754c195e nodeName:}" failed. No retries permitted until 2026-07-28 20:48:33.491977323 +0000 UTC m=+68.716320288 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/100dd861-fcc1-45da-ac5b-0797754c195e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7vtw8" (UID: "100dd861-fcc1-45da-ac5b-0797754c195e") : configmap "kube-rbac-proxy" not found
Jul 28 20:48:29.492062 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:29.492009    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls podName:100dd861-fcc1-45da-ac5b-0797754c195e nodeName:}" failed. No retries permitted until 2026-07-28 20:48:33.491996262 +0000 UTC m=+68.716339230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7vtw8" (UID: "100dd861-fcc1-45da-ac5b-0797754c195e") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:31.003054 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:31.003000    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret\") pod \"global-pull-secret-syncer-d8rwp\" (UID: \"f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22\") " pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:48:31.005563 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:31.005545    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 20:48:31.015540 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:31.015500    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22-original-pull-secret\") pod \"global-pull-secret-syncer-d8rwp\" (UID: \"f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22\") " pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:48:31.038674 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:31.038624    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d8rwp"
Jul 28 20:48:31.103459 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:31.103428    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnzhq\" (UniqueName: \"kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq\") pod \"network-check-target-qkd55\" (UID: \"ba39afab-1128-4d15-a2ab-bfe072b7eff1\") " pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:48:31.104090 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:31.103492    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs\") pod \"network-metrics-daemon-jtxnw\" (UID: \"6c1be7ce-d683-4505-b781-5f739d0d82b5\") " pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:48:31.106433 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:31.106409    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 20:48:31.106608 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:31.106595    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 20:48:31.114084 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:31.114059    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 20:48:31.114205 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:31.114139    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs podName:6c1be7ce-d683-4505-b781-5f739d0d82b5 nodeName:}" failed. No retries permitted until 2026-07-28 20:49:35.114116852 +0000 UTC m=+130.338459825 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs") pod "network-metrics-daemon-jtxnw" (UID: "6c1be7ce-d683-4505-b781-5f739d0d82b5") : secret "metrics-daemon-secret" not found
Jul 28 20:48:31.116131 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:31.116111    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 20:48:31.127501 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:31.127464    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnzhq\" (UniqueName: \"kubernetes.io/projected/ba39afab-1128-4d15-a2ab-bfe072b7eff1-kube-api-access-fnzhq\") pod \"network-check-target-qkd55\" (UID: \"ba39afab-1128-4d15-a2ab-bfe072b7eff1\") " pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:48:31.166739 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:31.166681    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-cgkpx" podStartSLOduration=4.17850659 podStartE2EDuration="5.166661791s" totalImagesPullingTime="988.155201ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:48:28.617734065 +0000 UTC m=+63.842077052" watchObservedRunningTime="2026-07-28 20:48:31.166661791 +0000 UTC m=+66.391004777"
Jul 28 20:48:31.167169 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:31.167151    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d8rwp"]
Jul 28 20:48:31.170369 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:48:31.170342    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b9d1d7_b8e8_4394_8d05_9bbc3ef89f22.slice/crio-92b51f678f8e7480d6989ec9b59d2c12b7366fe9d393bb6c016562b197b58121 WatchSource:0}: Error finding container 92b51f678f8e7480d6989ec9b59d2c12b7366fe9d393bb6c016562b197b58121: Status 404 returned error can't find the container with id 92b51f678f8e7480d6989ec9b59d2c12b7366fe9d393bb6c016562b197b58121
Jul 28 20:48:31.347873 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:31.347786    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j52pw\""
Jul 28 20:48:31.356474 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:31.356444    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:48:31.497080 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:31.497037    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qkd55"]
Jul 28 20:48:31.500677 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:48:31.500622    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba39afab_1128_4d15_a2ab_bfe072b7eff1.slice/crio-67d9f7ecc1c766ffb2bfcc721c5b32fdb781e27e5c80f0bd3e7e9da317ea5c83 WatchSource:0}: Error finding container 67d9f7ecc1c766ffb2bfcc721c5b32fdb781e27e5c80f0bd3e7e9da317ea5c83: Status 404 returned error can't find the container with id 67d9f7ecc1c766ffb2bfcc721c5b32fdb781e27e5c80f0bd3e7e9da317ea5c83
Jul 28 20:48:31.506626 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:31.506592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls\") pod \"dns-default-x6qn9\" (UID: \"41a72d4d-a128-4ed4-934f-0aebbe9825b6\") " pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:48:31.506788 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:31.506766    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 20:48:31.506865 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:31.506853    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls podName:41a72d4d-a128-4ed4-934f-0aebbe9825b6 nodeName:}" failed. No retries permitted until 2026-07-28 20:49:03.506832847 +0000 UTC m=+98.731175817 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls") pod "dns-default-x6qn9" (UID: "41a72d4d-a128-4ed4-934f-0aebbe9825b6") : secret "dns-default-metrics-tls" not found
Jul 28 20:48:31.607225 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:31.607128    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qkd55" event={"ID":"ba39afab-1128-4d15-a2ab-bfe072b7eff1","Type":"ContainerStarted","Data":"67d9f7ecc1c766ffb2bfcc721c5b32fdb781e27e5c80f0bd3e7e9da317ea5c83"}
Jul 28 20:48:31.608512 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:31.608483    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d8rwp" event={"ID":"f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22","Type":"ContainerStarted","Data":"92b51f678f8e7480d6989ec9b59d2c12b7366fe9d393bb6c016562b197b58121"}
Jul 28 20:48:33.525795 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:33.525747    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/100dd861-fcc1-45da-ac5b-0797754c195e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:33.526425 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:33.525864    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:33.526425 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:33.525900    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 20:48:33.526425 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:33.525984    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/100dd861-fcc1-45da-ac5b-0797754c195e-kube-rbac-proxy-cm podName:100dd861-fcc1-45da-ac5b-0797754c195e nodeName:}" failed. No retries permitted until 2026-07-28 20:48:41.525962527 +0000 UTC m=+76.750305513 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/100dd861-fcc1-45da-ac5b-0797754c195e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7vtw8" (UID: "100dd861-fcc1-45da-ac5b-0797754c195e") : configmap "kube-rbac-proxy" not found
Jul 28 20:48:33.526425 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:33.525999    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:33.526425 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:33.526046    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls podName:100dd861-fcc1-45da-ac5b-0797754c195e nodeName:}" failed. No retries permitted until 2026-07-28 20:48:41.526030924 +0000 UTC m=+76.750373891 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7vtw8" (UID: "100dd861-fcc1-45da-ac5b-0797754c195e") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:36.624612 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:36.624569    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qkd55" event={"ID":"ba39afab-1128-4d15-a2ab-bfe072b7eff1","Type":"ContainerStarted","Data":"bf25b211c8bb243d477532cad1e554d781c109dbb0f5b7510b664e40243f94d9"}
Jul 28 20:48:36.625105 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:36.624691    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:48:36.626167 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:36.626145    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d8rwp" event={"ID":"f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22","Type":"ContainerStarted","Data":"ca8e8ead437f369727b9fe1d5cc001d9411b8d39acb731dd151b90118e68730c"}
Jul 28 20:48:36.644516 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:36.644449    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qkd55" podStartSLOduration=67.235118404 podStartE2EDuration="1m11.644434266s" totalImagesPullingTime="4.409315862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:48:36.643547215 +0000 UTC m=+71.867890200" watchObservedRunningTime="2026-07-28 20:48:36.644434266 +0000 UTC m=+71.868777289"
Jul 28 20:48:36.661222 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:36.661170    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-d8rwp" podStartSLOduration=66.921267791 podStartE2EDuration="1m11.661153727s" totalImagesPullingTime="4.739885936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:48:36.660202391 +0000 UTC m=+71.884545379" watchObservedRunningTime="2026-07-28 20:48:36.661153727 +0000 UTC m=+71.885496713"
Jul 28 20:48:36.755744 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:36.755701    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-rvtwd\" (UID: \"18a1dfd7-9d6b-4bb8-a22c-956fca90189d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd"
Jul 28 20:48:36.755913 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:36.755803    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 20:48:36.755913 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:36.755858    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls podName:18a1dfd7-9d6b-4bb8-a22c-956fca90189d nodeName:}" failed. No retries permitted until 2026-07-28 20:48:52.755842597 +0000 UTC m=+87.980185562 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-rvtwd" (UID: "18a1dfd7-9d6b-4bb8-a22c-956fca90189d") : secret "cluster-monitoring-operator-tls" not found
Jul 28 20:48:36.857017 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:36.856981    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:36.859334 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:36.859306    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls\") pod \"image-registry-bb96c4ff7-hbx9q\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") " pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:36.990857 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:36.990823    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-52n7n\""
Jul 28 20:48:36.998837 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:36.998807    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:37.132650 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:37.132598    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bb96c4ff7-hbx9q"]
Jul 28 20:48:37.135590 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:48:37.135556    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda635f930_2655_4762_b3f6_67e2f947dc8c.slice/crio-9719abfa6ca72e65b536d45de30497c4c01fc07885aff324667e23b4f63244d2 WatchSource:0}: Error finding container 9719abfa6ca72e65b536d45de30497c4c01fc07885aff324667e23b4f63244d2: Status 404 returned error can't find the container with id 9719abfa6ca72e65b536d45de30497c4c01fc07885aff324667e23b4f63244d2
Jul 28 20:48:37.631247 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:37.631212    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q" event={"ID":"a635f930-2655-4762-b3f6-67e2f947dc8c","Type":"ContainerStarted","Data":"570b8d8ef08ab9cbd92c78c1a9a3fe65c9c26ae9013940407958bfaf636cf71e"}
Jul 28 20:48:37.631649 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:37.631264    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q" event={"ID":"a635f930-2655-4762-b3f6-67e2f947dc8c","Type":"ContainerStarted","Data":"9719abfa6ca72e65b536d45de30497c4c01fc07885aff324667e23b4f63244d2"}
Jul 28 20:48:37.655876 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:37.655805    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q" podStartSLOduration=16.655783724 podStartE2EDuration="16.655783724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:48:37.653813742 +0000 UTC m=+72.878156727" watchObservedRunningTime="2026-07-28 20:48:37.655783724 +0000 UTC m=+72.880126711"
Jul 28 20:48:38.634240 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:38.634213    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:48:41.598579 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:41.598537    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/100dd861-fcc1-45da-ac5b-0797754c195e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:41.599002 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:41.598620    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:41.599002 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:41.598739    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:41.599002 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:48:41.598793    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls podName:100dd861-fcc1-45da-ac5b-0797754c195e nodeName:}" failed. No retries permitted until 2026-07-28 20:48:57.598777419 +0000 UTC m=+92.823120383 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7vtw8" (UID: "100dd861-fcc1-45da-ac5b-0797754c195e") : secret "insights-runtime-extractor-tls" not found
Jul 28 20:48:41.599129 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:41.599029    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/100dd861-fcc1-45da-ac5b-0797754c195e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:52.780985 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:52.780852    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-rvtwd\" (UID: \"18a1dfd7-9d6b-4bb8-a22c-956fca90189d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd"
Jul 28 20:48:52.783259 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:52.783238    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a1dfd7-9d6b-4bb8-a22c-956fca90189d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-rvtwd\" (UID: \"18a1dfd7-9d6b-4bb8-a22c-956fca90189d\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd"
Jul 28 20:48:53.042843 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:53.042747    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-sr5d7\""
Jul 28 20:48:53.050532 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:53.050502    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd"
Jul 28 20:48:53.181887 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:53.181857    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd"]
Jul 28 20:48:53.184972 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:48:53.184947    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a1dfd7_9d6b_4bb8_a22c_956fca90189d.slice/crio-eccc4a3716983650d458c2689cd9c6b5394921d51b040aad0ba58b01b269fe07 WatchSource:0}: Error finding container eccc4a3716983650d458c2689cd9c6b5394921d51b040aad0ba58b01b269fe07: Status 404 returned error can't find the container with id eccc4a3716983650d458c2689cd9c6b5394921d51b040aad0ba58b01b269fe07
Jul 28 20:48:53.679369 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:53.679329    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd" event={"ID":"18a1dfd7-9d6b-4bb8-a22c-956fca90189d","Type":"ContainerStarted","Data":"eccc4a3716983650d458c2689cd9c6b5394921d51b040aad0ba58b01b269fe07"}
Jul 28 20:48:54.524962 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.524917    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-wsvnr"]
Jul 28 20:48:54.529407 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.529376    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7866c949f5-c54f4"]
Jul 28 20:48:54.529555 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.529511    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-wsvnr"
Jul 28 20:48:54.531913 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.531856    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 20:48:54.531913 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.531895    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 20:48:54.532406 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.532378    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-qpkck\""
Jul 28 20:48:54.532525 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.532513    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7866c949f5-c54f4"
Jul 28 20:48:54.534788 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.534768    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 20:48:54.535383 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.535363    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 20:48:54.535833 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.535815    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 20:48:54.537077 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.536909    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 20:48:54.542417 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.542351    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7866c949f5-c54f4"]
Jul 28 20:48:54.543528 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.543507    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-wsvnr"]
Jul 28 20:48:54.554945 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.554920    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bb96c4ff7-hbx9q"]
Jul 28 20:48:54.560357 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.560329    2565 patch_prober.go:28] interesting pod/image-registry-bb96c4ff7-hbx9q container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 28 20:48:54.560475 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.560390    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q" podUID="a635f930-2655-4762-b3f6-67e2f947dc8c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 20:48:54.594894 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.594860    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c15ac9e-5a4c-4110-93d8-4217a56ea81b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-wsvnr\" (UID: \"4c15ac9e-5a4c-4110-93d8-4217a56ea81b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-wsvnr"
Jul 28 20:48:54.594894 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.594897    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4c15ac9e-5a4c-4110-93d8-4217a56ea81b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-wsvnr\" (UID: \"4c15ac9e-5a4c-4110-93d8-4217a56ea81b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-wsvnr"
Jul 28 20:48:54.613896 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.613856    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-6bkcp"]
Jul 28 20:48:54.617116 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.617092    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-586cdbddd-fbphg"]
Jul 28 20:48:54.617269 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.617238    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-6bkcp"
Jul 28 20:48:54.619052 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.619031    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 20:48:54.619173 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.619033    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 20:48:54.619462 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.619444    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-ggvgs\""
Jul 28 20:48:54.620194 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.620175    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.634145 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.634118    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-6bkcp"]
Jul 28 20:48:54.634992 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.634969    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-586cdbddd-fbphg"]
Jul 28 20:48:54.696257 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.696211    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ff0086b-af29-49c2-9569-ee3b382929b3-bound-sa-token\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.696441 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.696277    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1ff0086b-af29-49c2-9569-ee3b382929b3-trusted-ca\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.696441 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.696385    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1ff0086b-af29-49c2-9569-ee3b382929b3-image-registry-private-configuration\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.696441 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.696423    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1ff0086b-af29-49c2-9569-ee3b382929b3-registry-certificates\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.696572 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.696489    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4c15ac9e-5a4c-4110-93d8-4217a56ea81b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-wsvnr\" (UID: \"4c15ac9e-5a4c-4110-93d8-4217a56ea81b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-wsvnr"
Jul 28 20:48:54.696572 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.696520    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29d9d6a7-540c-46a8-81f0-17169b81eccf-tmp\") pod \"klusterlet-addon-workmgr-7866c949f5-c54f4\" (UID: \"29d9d6a7-540c-46a8-81f0-17169b81eccf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7866c949f5-c54f4"
Jul 28 20:48:54.696572 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.696557    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1ff0086b-af29-49c2-9569-ee3b382929b3-installation-pull-secrets\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.696761 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.696593    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1ff0086b-af29-49c2-9569-ee3b382929b3-registry-tls\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.696761 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.696621    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp6w8\" (UniqueName: \"kubernetes.io/projected/eb524232-3306-4310-968b-178c376138e8-kube-api-access-lp6w8\") pod \"downloads-7c97dc4555-6bkcp\" (UID: \"eb524232-3306-4310-968b-178c376138e8\") " pod="openshift-console/downloads-7c97dc4555-6bkcp"
Jul 28 20:48:54.696761 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.696732    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dglnz\" (UniqueName: \"kubernetes.io/projected/29d9d6a7-540c-46a8-81f0-17169b81eccf-kube-api-access-dglnz\") pod \"klusterlet-addon-workmgr-7866c949f5-c54f4\" (UID: \"29d9d6a7-540c-46a8-81f0-17169b81eccf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7866c949f5-c54f4"
Jul 28 20:48:54.696908 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.696770    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/29d9d6a7-540c-46a8-81f0-17169b81eccf-klusterlet-config\") pod \"klusterlet-addon-workmgr-7866c949f5-c54f4\" (UID: \"29d9d6a7-540c-46a8-81f0-17169b81eccf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7866c949f5-c54f4"
Jul 28 20:48:54.696908 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.696803    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c15ac9e-5a4c-4110-93d8-4217a56ea81b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-wsvnr\" (UID: \"4c15ac9e-5a4c-4110-93d8-4217a56ea81b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-wsvnr"
Jul 28 20:48:54.696908 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.696828    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1ff0086b-af29-49c2-9569-ee3b382929b3-ca-trust-extracted\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.696908 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.696854    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj7bc\" (UniqueName: \"kubernetes.io/projected/1ff0086b-af29-49c2-9569-ee3b382929b3-kube-api-access-rj7bc\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.697316 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.697294    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4c15ac9e-5a4c-4110-93d8-4217a56ea81b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-wsvnr\" (UID: \"4c15ac9e-5a4c-4110-93d8-4217a56ea81b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-wsvnr"
Jul 28 20:48:54.699455 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.699434    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c15ac9e-5a4c-4110-93d8-4217a56ea81b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-wsvnr\" (UID: \"4c15ac9e-5a4c-4110-93d8-4217a56ea81b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-wsvnr"
Jul 28 20:48:54.797935 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.797833    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1ff0086b-af29-49c2-9569-ee3b382929b3-ca-trust-extracted\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.797935 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.797890    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj7bc\" (UniqueName: \"kubernetes.io/projected/1ff0086b-af29-49c2-9569-ee3b382929b3-kube-api-access-rj7bc\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.797935 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.797936    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ff0086b-af29-49c2-9569-ee3b382929b3-bound-sa-token\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.798210 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.797986    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1ff0086b-af29-49c2-9569-ee3b382929b3-trusted-ca\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.798210 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.798029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1ff0086b-af29-49c2-9569-ee3b382929b3-image-registry-private-configuration\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.798210 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.798056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1ff0086b-af29-49c2-9569-ee3b382929b3-registry-certificates\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.798210 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.798108    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29d9d6a7-540c-46a8-81f0-17169b81eccf-tmp\") pod \"klusterlet-addon-workmgr-7866c949f5-c54f4\" (UID: \"29d9d6a7-540c-46a8-81f0-17169b81eccf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7866c949f5-c54f4"
Jul 28 20:48:54.798210 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.798137    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1ff0086b-af29-49c2-9569-ee3b382929b3-installation-pull-secrets\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.798210 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.798169    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1ff0086b-af29-49c2-9569-ee3b382929b3-registry-tls\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.798210 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.798197    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp6w8\" (UniqueName: \"kubernetes.io/projected/eb524232-3306-4310-968b-178c376138e8-kube-api-access-lp6w8\") pod \"downloads-7c97dc4555-6bkcp\" (UID: \"eb524232-3306-4310-968b-178c376138e8\") " pod="openshift-console/downloads-7c97dc4555-6bkcp"
Jul 28 20:48:54.798540 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.798232    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dglnz\" (UniqueName: \"kubernetes.io/projected/29d9d6a7-540c-46a8-81f0-17169b81eccf-kube-api-access-dglnz\") pod \"klusterlet-addon-workmgr-7866c949f5-c54f4\" (UID: \"29d9d6a7-540c-46a8-81f0-17169b81eccf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7866c949f5-c54f4"
Jul 28 20:48:54.798540 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.798259    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/29d9d6a7-540c-46a8-81f0-17169b81eccf-klusterlet-config\") pod \"klusterlet-addon-workmgr-7866c949f5-c54f4\" (UID: \"29d9d6a7-540c-46a8-81f0-17169b81eccf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7866c949f5-c54f4"
Jul 28 20:48:54.798540 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.798317    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1ff0086b-af29-49c2-9569-ee3b382929b3-ca-trust-extracted\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.798717 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.798541    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29d9d6a7-540c-46a8-81f0-17169b81eccf-tmp\") pod \"klusterlet-addon-workmgr-7866c949f5-c54f4\" (UID: \"29d9d6a7-540c-46a8-81f0-17169b81eccf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7866c949f5-c54f4"
Jul 28 20:48:54.799147 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.799117    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1ff0086b-af29-49c2-9569-ee3b382929b3-trusted-ca\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.799308 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.799284    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1ff0086b-af29-49c2-9569-ee3b382929b3-registry-certificates\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.801056 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.801030    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1ff0086b-af29-49c2-9569-ee3b382929b3-installation-pull-secrets\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.801140 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.801120    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1ff0086b-af29-49c2-9569-ee3b382929b3-image-registry-private-configuration\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.801318 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.801298    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1ff0086b-af29-49c2-9569-ee3b382929b3-registry-tls\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.801359 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.801336    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/29d9d6a7-540c-46a8-81f0-17169b81eccf-klusterlet-config\") pod \"klusterlet-addon-workmgr-7866c949f5-c54f4\" (UID: \"29d9d6a7-540c-46a8-81f0-17169b81eccf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7866c949f5-c54f4"
Jul 28 20:48:54.807024 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.806994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ff0086b-af29-49c2-9569-ee3b382929b3-bound-sa-token\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.807934 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.807913    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp6w8\" (UniqueName: \"kubernetes.io/projected/eb524232-3306-4310-968b-178c376138e8-kube-api-access-lp6w8\") pod \"downloads-7c97dc4555-6bkcp\" (UID: \"eb524232-3306-4310-968b-178c376138e8\") " pod="openshift-console/downloads-7c97dc4555-6bkcp"
Jul 28 20:48:54.808245 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.808228    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj7bc\" (UniqueName: \"kubernetes.io/projected/1ff0086b-af29-49c2-9569-ee3b382929b3-kube-api-access-rj7bc\") pod \"image-registry-586cdbddd-fbphg\" (UID: \"1ff0086b-af29-49c2-9569-ee3b382929b3\") " pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.808375 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.808346    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dglnz\" (UniqueName: \"kubernetes.io/projected/29d9d6a7-540c-46a8-81f0-17169b81eccf-kube-api-access-dglnz\") pod \"klusterlet-addon-workmgr-7866c949f5-c54f4\" (UID: \"29d9d6a7-540c-46a8-81f0-17169b81eccf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7866c949f5-c54f4"
Jul 28 20:48:54.843326 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.843290    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-wsvnr"
Jul 28 20:48:54.850745 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.850723    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7866c949f5-c54f4"
Jul 28 20:48:54.935057 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.935031    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-6bkcp"
Jul 28 20:48:54.940933 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.940550    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:54.999390 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:54.998550    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-wsvnr"]
Jul 28 20:48:55.009439 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:48:55.009392    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c15ac9e_5a4c_4110_93d8_4217a56ea81b.slice/crio-b0cfd4739f875719bc7a3023c5efbf654ecf6ef1dc2a56798b511ffa3dfe26db WatchSource:0}: Error finding container b0cfd4739f875719bc7a3023c5efbf654ecf6ef1dc2a56798b511ffa3dfe26db: Status 404 returned error can't find the container with id b0cfd4739f875719bc7a3023c5efbf654ecf6ef1dc2a56798b511ffa3dfe26db
Jul 28 20:48:55.023182 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:55.023096    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7866c949f5-c54f4"]
Jul 28 20:48:55.027563 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:48:55.027530    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d9d6a7_540c_46a8_81f0_17169b81eccf.slice/crio-7bccc1ba35a0b271b15830ad1c2b7dd8f8346582e73ea968730189f3ad943eae WatchSource:0}: Error finding container 7bccc1ba35a0b271b15830ad1c2b7dd8f8346582e73ea968730189f3ad943eae: Status 404 returned error can't find the container with id 7bccc1ba35a0b271b15830ad1c2b7dd8f8346582e73ea968730189f3ad943eae
Jul 28 20:48:55.106253 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:55.106097    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-586cdbddd-fbphg"]
Jul 28 20:48:55.109215 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:48:55.109189    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff0086b_af29_49c2_9569_ee3b382929b3.slice/crio-ff9a4942c1cf4ba750ae3174d6464fcbb357d98172d883af5cf70bc4a9b5b2a7 WatchSource:0}: Error finding container ff9a4942c1cf4ba750ae3174d6464fcbb357d98172d883af5cf70bc4a9b5b2a7: Status 404 returned error can't find the container with id ff9a4942c1cf4ba750ae3174d6464fcbb357d98172d883af5cf70bc4a9b5b2a7
Jul 28 20:48:55.112127 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:55.112032    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-6bkcp"]
Jul 28 20:48:55.690963 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:55.690920    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7866c949f5-c54f4" event={"ID":"29d9d6a7-540c-46a8-81f0-17169b81eccf","Type":"ContainerStarted","Data":"7bccc1ba35a0b271b15830ad1c2b7dd8f8346582e73ea968730189f3ad943eae"}
Jul 28 20:48:55.695397 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:55.694511    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-586cdbddd-fbphg" event={"ID":"1ff0086b-af29-49c2-9569-ee3b382929b3","Type":"ContainerStarted","Data":"b052315b4c9edcfb45e7ed932f1b58d5b850776e133c6204dbdf8173628b4815"}
Jul 28 20:48:55.695397 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:55.694565    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-586cdbddd-fbphg" event={"ID":"1ff0086b-af29-49c2-9569-ee3b382929b3","Type":"ContainerStarted","Data":"ff9a4942c1cf4ba750ae3174d6464fcbb357d98172d883af5cf70bc4a9b5b2a7"}
Jul 28 20:48:55.695397 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:55.695365    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:48:55.697332 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:55.697303    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd" event={"ID":"18a1dfd7-9d6b-4bb8-a22c-956fca90189d","Type":"ContainerStarted","Data":"45fe67a22bd46f136ca199db61f3e4712da174dfa570e9c0f1a0aa7b8aadb5b7"}
Jul 28 20:48:55.706497 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:55.699975    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-6bkcp" event={"ID":"eb524232-3306-4310-968b-178c376138e8","Type":"ContainerStarted","Data":"86b2a0ec2fab3c3fda95db0de39ab492fd59586e6caff4cc6bf3220c73336722"}
Jul 28 20:48:55.706497 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:55.702035    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-wsvnr" event={"ID":"4c15ac9e-5a4c-4110-93d8-4217a56ea81b","Type":"ContainerStarted","Data":"b0cfd4739f875719bc7a3023c5efbf654ecf6ef1dc2a56798b511ffa3dfe26db"}
Jul 28 20:48:55.718226 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:55.717104    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-586cdbddd-fbphg" podStartSLOduration=1.717081928 podStartE2EDuration="1.717081928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:48:55.716147151 +0000 UTC m=+90.940490150" watchObservedRunningTime="2026-07-28 20:48:55.717081928 +0000 UTC m=+90.941424894"
Jul 28 20:48:56.709507 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:56.709463    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-wsvnr" event={"ID":"4c15ac9e-5a4c-4110-93d8-4217a56ea81b","Type":"ContainerStarted","Data":"9af801c8e227237ac5bb00c69e3977a3b157e6f7c892bd06f46cd7c5e3c884be"}
Jul 28 20:48:56.732150 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:56.731093    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-rvtwd" podStartSLOduration=35.008441071 podStartE2EDuration="36.731071102s" totalImagesPullingTime="1.722630031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:48:55.735483378 +0000 UTC m=+90.959826363" watchObservedRunningTime="2026-07-28 20:48:56.731071102 +0000 UTC m=+91.955414090"
Jul 28 20:48:56.732150 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:56.731711    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-wsvnr" podStartSLOduration=1.452380424 podStartE2EDuration="2.731698294s" totalImagesPullingTime="1.27931787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:48:56.730424834 +0000 UTC m=+91.954767823" watchObservedRunningTime="2026-07-28 20:48:56.731698294 +0000 UTC m=+91.956041280"
Jul 28 20:48:57.630458 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:57.630415    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:57.633630 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:57.633602    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/100dd861-fcc1-45da-ac5b-0797754c195e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vtw8\" (UID: \"100dd861-fcc1-45da-ac5b-0797754c195e\") " pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:57.645670 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:57.645530    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7vtw8"
Jul 28 20:48:58.660972 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:58.660879    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7vtw8"]
Jul 28 20:48:58.666110 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:48:58.666077    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100dd861_fcc1_45da_ac5b_0797754c195e.slice/crio-05052527c56906b9a0cbc19dd495703194575bcc5b406800641234934aa8b0ac WatchSource:0}: Error finding container 05052527c56906b9a0cbc19dd495703194575bcc5b406800641234934aa8b0ac: Status 404 returned error can't find the container with id 05052527c56906b9a0cbc19dd495703194575bcc5b406800641234934aa8b0ac
Jul 28 20:48:58.723227 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:48:58.723182    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7vtw8" event={"ID":"100dd861-fcc1-45da-ac5b-0797754c195e","Type":"ContainerStarted","Data":"05052527c56906b9a0cbc19dd495703194575bcc5b406800641234934aa8b0ac"}
Jul 28 20:49:00.546391 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.546356    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5974c79874-wcg9q"]
Jul 28 20:49:00.549349 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.549322    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.552668 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.552625    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 20:49:00.552823 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.552799    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 20:49:00.552952 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.552930    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 20:49:00.553090 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.553019    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 20:49:00.553306 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.553271    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wj57k\""
Jul 28 20:49:00.553395 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.553356    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 20:49:00.562322 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.561427    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 20:49:00.564250 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.564225    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5974c79874-wcg9q"]
Jul 28 20:49:00.655529 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.655123    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-console-serving-cert\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.655529 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.655175    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-console-config\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.655529 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.655209    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-trusted-ca-bundle\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.655529 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.655274    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-oauth-serving-cert\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.655529 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.655302    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kmsm\" (UniqueName: \"kubernetes.io/projected/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-kube-api-access-5kmsm\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.655529 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.655375    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-console-oauth-config\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.655529 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.655398    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-service-ca\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.734111 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.734069    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7866c949f5-c54f4" event={"ID":"29d9d6a7-540c-46a8-81f0-17169b81eccf","Type":"ContainerStarted","Data":"407dc44e38d84375c0c78aa302541de504323b695eb5c5d46b0565aef7c3b2a1"}
Jul 28 20:49:00.734443 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.734421    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7866c949f5-c54f4"
Jul 28 20:49:00.736340 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.736304    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7866c949f5-c54f4"
Jul 28 20:49:00.751266 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.751198    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7866c949f5-c54f4" podStartSLOduration=1.696756069 podStartE2EDuration="6.751176499s" totalImagesPullingTime="5.05442043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:49:00.750787874 +0000 UTC m=+95.975130860" watchObservedRunningTime="2026-07-28 20:49:00.751176499 +0000 UTC m=+95.975519487"
Jul 28 20:49:00.756046 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.756010    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-console-serving-cert\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.756205 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.756056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-console-config\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.756205 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.756087    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-trusted-ca-bundle\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.756205 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.756138    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-oauth-serving-cert\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.756205 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.756166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kmsm\" (UniqueName: \"kubernetes.io/projected/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-kube-api-access-5kmsm\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.756419 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.756350    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-console-oauth-config\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.756419 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.756374    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-service-ca\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.757588 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.757540    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-service-ca\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.757588 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.757555    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-console-config\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.757588 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.757563    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-oauth-serving-cert\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.757834 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.757620    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-trusted-ca-bundle\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.758982 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.758937    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-console-serving-cert\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.760001 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.759974    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-console-oauth-config\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.766394 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.766369    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kmsm\" (UniqueName: \"kubernetes.io/projected/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-kube-api-access-5kmsm\") pod \"console-5974c79874-wcg9q\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") " pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:00.865177 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:00.865087    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:01.053386 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:01.053350    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5974c79874-wcg9q"]
Jul 28 20:49:01.057629 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:49:01.057594    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecc2a5f_ae06_4925_bc69_775b0f422a7f.slice/crio-cd826be368c6d3dadc258cac8f0ce5e3fc3d3eacfe22bea3e07745db76b13e84 WatchSource:0}: Error finding container cd826be368c6d3dadc258cac8f0ce5e3fc3d3eacfe22bea3e07745db76b13e84: Status 404 returned error can't find the container with id cd826be368c6d3dadc258cac8f0ce5e3fc3d3eacfe22bea3e07745db76b13e84
Jul 28 20:49:01.741195 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:01.741150    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5974c79874-wcg9q" event={"ID":"7ecc2a5f-ae06-4925-bc69-775b0f422a7f","Type":"ContainerStarted","Data":"cd826be368c6d3dadc258cac8f0ce5e3fc3d3eacfe22bea3e07745db76b13e84"}
Jul 28 20:49:03.583583 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:03.583399    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls\") pod \"dns-default-x6qn9\" (UID: \"41a72d4d-a128-4ed4-934f-0aebbe9825b6\") " pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:49:03.586593 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:03.586557    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41a72d4d-a128-4ed4-934f-0aebbe9825b6-metrics-tls\") pod \"dns-default-x6qn9\" (UID: \"41a72d4d-a128-4ed4-934f-0aebbe9825b6\") " pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:49:03.825204 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:03.825157    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ghnpw\""
Jul 28 20:49:03.833382 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:03.833299    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:49:04.561445 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:04.561032    2565 patch_prober.go:28] interesting pod/image-registry-bb96c4ff7-hbx9q container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 28 20:49:04.561445 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:04.561096    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q" podUID="a635f930-2655-4762-b3f6-67e2f947dc8c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 20:49:07.634106 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:07.634072    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qkd55"
Jul 28 20:49:11.007299 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.007254    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb"]
Jul 28 20:49:11.010845 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.010816    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb"
Jul 28 20:49:11.012920 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.012891    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 20:49:11.013063 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.012946    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 20:49:11.013063 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.012998    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9mmv5\""
Jul 28 20:49:11.013367 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.013347    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 20:49:11.020253 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.020146    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb"]
Jul 28 20:49:11.049290 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.049242    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6525614c-96ff-4222-83eb-a492804d204a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-42cwb\" (UID: \"6525614c-96ff-4222-83eb-a492804d204a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb"
Jul 28 20:49:11.049485 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.049323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqbjn\" (UniqueName: \"kubernetes.io/projected/6525614c-96ff-4222-83eb-a492804d204a-kube-api-access-sqbjn\") pod \"openshift-state-metrics-b8f49f5c6-42cwb\" (UID: \"6525614c-96ff-4222-83eb-a492804d204a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb"
Jul 28 20:49:11.049485 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.049355    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6525614c-96ff-4222-83eb-a492804d204a-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-42cwb\" (UID: \"6525614c-96ff-4222-83eb-a492804d204a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb"
Jul 28 20:49:11.049485 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.049393    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6525614c-96ff-4222-83eb-a492804d204a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-42cwb\" (UID: \"6525614c-96ff-4222-83eb-a492804d204a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb"
Jul 28 20:49:11.076665 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.076612    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t5fvv"]
Jul 28 20:49:11.079584 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.079561    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.081927 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.081904    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 20:49:11.083155 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.083105    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-d2fw6\""
Jul 28 20:49:11.083904 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.083883    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 20:49:11.088492 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.088471    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 20:49:11.150060 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.150027    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc9gt\" (UniqueName: \"kubernetes.io/projected/3568e7d8-5fff-4276-83d3-b2d9400d4d40-kube-api-access-sc9gt\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.150242 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.150066    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6525614c-96ff-4222-83eb-a492804d204a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-42cwb\" (UID: \"6525614c-96ff-4222-83eb-a492804d204a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb"
Jul 28 20:49:11.150242 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.150113    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3568e7d8-5fff-4276-83d3-b2d9400d4d40-node-exporter-textfile\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.150242 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.150141    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3568e7d8-5fff-4276-83d3-b2d9400d4d40-node-exporter-accelerators-collector-config\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.150242 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.150176    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqbjn\" (UniqueName: \"kubernetes.io/projected/6525614c-96ff-4222-83eb-a492804d204a-kube-api-access-sqbjn\") pod \"openshift-state-metrics-b8f49f5c6-42cwb\" (UID: \"6525614c-96ff-4222-83eb-a492804d204a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb"
Jul 28 20:49:11.150242 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:49:11.150222    2565 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 28 20:49:11.150242 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.150223    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3568e7d8-5fff-4276-83d3-b2d9400d4d40-metrics-client-ca\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.150515 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.150397    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3568e7d8-5fff-4276-83d3-b2d9400d4d40-sys\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.150515 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:49:11.150413    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6525614c-96ff-4222-83eb-a492804d204a-openshift-state-metrics-tls podName:6525614c-96ff-4222-83eb-a492804d204a nodeName:}" failed. No retries permitted until 2026-07-28 20:49:11.650390337 +0000 UTC m=+106.874733310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6525614c-96ff-4222-83eb-a492804d204a-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-42cwb" (UID: "6525614c-96ff-4222-83eb-a492804d204a") : secret "openshift-state-metrics-tls" not found
Jul 28 20:49:11.150515 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.150450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6525614c-96ff-4222-83eb-a492804d204a-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-42cwb\" (UID: \"6525614c-96ff-4222-83eb-a492804d204a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb"
Jul 28 20:49:11.150515 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.150485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6525614c-96ff-4222-83eb-a492804d204a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-42cwb\" (UID: \"6525614c-96ff-4222-83eb-a492804d204a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb"
Jul 28 20:49:11.150759 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.150529    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3568e7d8-5fff-4276-83d3-b2d9400d4d40-root\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.150759 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.150572    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3568e7d8-5fff-4276-83d3-b2d9400d4d40-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.150759 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.150670    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3568e7d8-5fff-4276-83d3-b2d9400d4d40-node-exporter-wtmp\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.150759 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.150704    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3568e7d8-5fff-4276-83d3-b2d9400d4d40-node-exporter-tls\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.151534 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.151511    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6525614c-96ff-4222-83eb-a492804d204a-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-42cwb\" (UID: \"6525614c-96ff-4222-83eb-a492804d204a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb"
Jul 28 20:49:11.153233 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.153213    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6525614c-96ff-4222-83eb-a492804d204a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-42cwb\" (UID: \"6525614c-96ff-4222-83eb-a492804d204a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb"
Jul 28 20:49:11.163760 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.163732    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqbjn\" (UniqueName: \"kubernetes.io/projected/6525614c-96ff-4222-83eb-a492804d204a-kube-api-access-sqbjn\") pod \"openshift-state-metrics-b8f49f5c6-42cwb\" (UID: \"6525614c-96ff-4222-83eb-a492804d204a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb"
Jul 28 20:49:11.251946 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.251903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3568e7d8-5fff-4276-83d3-b2d9400d4d40-node-exporter-textfile\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.251946 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.251951    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3568e7d8-5fff-4276-83d3-b2d9400d4d40-node-exporter-accelerators-collector-config\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.252188 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.251987    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3568e7d8-5fff-4276-83d3-b2d9400d4d40-metrics-client-ca\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.252188 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.252021    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3568e7d8-5fff-4276-83d3-b2d9400d4d40-sys\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.252188 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.252094    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3568e7d8-5fff-4276-83d3-b2d9400d4d40-sys\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.252188 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.252130    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3568e7d8-5fff-4276-83d3-b2d9400d4d40-root\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.252188 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.252161    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3568e7d8-5fff-4276-83d3-b2d9400d4d40-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.252447 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.252215    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3568e7d8-5fff-4276-83d3-b2d9400d4d40-node-exporter-wtmp\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.252447 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.252244    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3568e7d8-5fff-4276-83d3-b2d9400d4d40-node-exporter-tls\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.252447 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.252256    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3568e7d8-5fff-4276-83d3-b2d9400d4d40-root\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.252447 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.252297    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc9gt\" (UniqueName: \"kubernetes.io/projected/3568e7d8-5fff-4276-83d3-b2d9400d4d40-kube-api-access-sc9gt\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.252447 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.252363    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3568e7d8-5fff-4276-83d3-b2d9400d4d40-node-exporter-textfile\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.252447 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.252413    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3568e7d8-5fff-4276-83d3-b2d9400d4d40-node-exporter-wtmp\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.252447 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:49:11.252447    2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 20:49:11.252810 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:49:11.252503    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3568e7d8-5fff-4276-83d3-b2d9400d4d40-node-exporter-tls podName:3568e7d8-5fff-4276-83d3-b2d9400d4d40 nodeName:}" failed. No retries permitted until 2026-07-28 20:49:11.752485516 +0000 UTC m=+106.976828485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3568e7d8-5fff-4276-83d3-b2d9400d4d40-node-exporter-tls") pod "node-exporter-t5fvv" (UID: "3568e7d8-5fff-4276-83d3-b2d9400d4d40") : secret "node-exporter-tls" not found
Jul 28 20:49:11.252810 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.252710    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3568e7d8-5fff-4276-83d3-b2d9400d4d40-metrics-client-ca\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.252810 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.252714    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3568e7d8-5fff-4276-83d3-b2d9400d4d40-node-exporter-accelerators-collector-config\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.254895 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.254870    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3568e7d8-5fff-4276-83d3-b2d9400d4d40-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.264950 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.264871    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc9gt\" (UniqueName: \"kubernetes.io/projected/3568e7d8-5fff-4276-83d3-b2d9400d4d40-kube-api-access-sc9gt\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.655923 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.655840    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6525614c-96ff-4222-83eb-a492804d204a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-42cwb\" (UID: \"6525614c-96ff-4222-83eb-a492804d204a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb"
Jul 28 20:49:11.658382 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.658356    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6525614c-96ff-4222-83eb-a492804d204a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-42cwb\" (UID: \"6525614c-96ff-4222-83eb-a492804d204a\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb"
Jul 28 20:49:11.756801 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.756751    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3568e7d8-5fff-4276-83d3-b2d9400d4d40-node-exporter-tls\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.759399 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.759361    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3568e7d8-5fff-4276-83d3-b2d9400d4d40-node-exporter-tls\") pod \"node-exporter-t5fvv\" (UID: \"3568e7d8-5fff-4276-83d3-b2d9400d4d40\") " pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:11.923915 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.923829    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb"
Jul 28 20:49:11.990180 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:11.990139    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t5fvv"
Jul 28 20:49:14.400086 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:49:14.400057    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3568e7d8_5fff_4276_83d3_b2d9400d4d40.slice/crio-3aa226e51d9f55ffcdf1829a0dfac29a13f31a9b13f9cc569db9f956c4a24a94 WatchSource:0}: Error finding container 3aa226e51d9f55ffcdf1829a0dfac29a13f31a9b13f9cc569db9f956c4a24a94: Status 404 returned error can't find the container with id 3aa226e51d9f55ffcdf1829a0dfac29a13f31a9b13f9cc569db9f956c4a24a94
Jul 28 20:49:14.561925 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.561883    2565 patch_prober.go:28] interesting pod/image-registry-bb96c4ff7-hbx9q container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 28 20:49:14.562107 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.561950    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q" podUID="a635f930-2655-4762-b3f6-67e2f947dc8c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 20:49:14.566737 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.566596    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x6qn9"]
Jul 28 20:49:14.569604 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:49:14.569562    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a72d4d_a128_4ed4_934f_0aebbe9825b6.slice/crio-e1e2242f0120ccb0ebd7210b62d5e44f900ac7bbd3f0aa7ff0cc148863a9a26d WatchSource:0}: Error finding container e1e2242f0120ccb0ebd7210b62d5e44f900ac7bbd3f0aa7ff0cc148863a9a26d: Status 404 returned error can't find the container with id e1e2242f0120ccb0ebd7210b62d5e44f900ac7bbd3f0aa7ff0cc148863a9a26d
Jul 28 20:49:14.597225 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.597189    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb"]
Jul 28 20:49:14.600820 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:49:14.600788    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6525614c_96ff_4222_83eb_a492804d204a.slice/crio-ea0a299b85a5c7586b177667791b223f37aebca67cab9e5cb30b22295be04219 WatchSource:0}: Error finding container ea0a299b85a5c7586b177667791b223f37aebca67cab9e5cb30b22295be04219: Status 404 returned error can't find the container with id ea0a299b85a5c7586b177667791b223f37aebca67cab9e5cb30b22295be04219
Jul 28 20:49:14.795246 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.795204    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb" event={"ID":"6525614c-96ff-4222-83eb-a492804d204a","Type":"ContainerStarted","Data":"4e79a408b421a668c59d1e93c5c14e58533c096981301bf1db6b6975544eed8a"}
Jul 28 20:49:14.795410 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.795283    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb" event={"ID":"6525614c-96ff-4222-83eb-a492804d204a","Type":"ContainerStarted","Data":"a0c2a563c1228eed873ab3389dd781fda88aabd55f70b0784fdb461860853d79"}
Jul 28 20:49:14.795410 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.795318    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb" event={"ID":"6525614c-96ff-4222-83eb-a492804d204a","Type":"ContainerStarted","Data":"ea0a299b85a5c7586b177667791b223f37aebca67cab9e5cb30b22295be04219"}
Jul 28 20:49:14.797038 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.796981    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5974c79874-wcg9q" event={"ID":"7ecc2a5f-ae06-4925-bc69-775b0f422a7f","Type":"ContainerStarted","Data":"b4089e44b6d5eaa5b47fe5af74a1e506ee8c906afff24e891242c0de4f74bfb0"}
Jul 28 20:49:14.798255 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.798219    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x6qn9" event={"ID":"41a72d4d-a128-4ed4-934f-0aebbe9825b6","Type":"ContainerStarted","Data":"e1e2242f0120ccb0ebd7210b62d5e44f900ac7bbd3f0aa7ff0cc148863a9a26d"}
Jul 28 20:49:14.799841 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.799756    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7vtw8" event={"ID":"100dd861-fcc1-45da-ac5b-0797754c195e","Type":"ContainerStarted","Data":"8ae089d76e2ad53975cc22b86855208ac0dd2757f97b474f6be46f4587eb9868"}
Jul 28 20:49:14.801079 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.801029    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t5fvv" event={"ID":"3568e7d8-5fff-4276-83d3-b2d9400d4d40","Type":"ContainerStarted","Data":"3aa226e51d9f55ffcdf1829a0dfac29a13f31a9b13f9cc569db9f956c4a24a94"}
Jul 28 20:49:14.802767 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.802746    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-6bkcp" event={"ID":"eb524232-3306-4310-968b-178c376138e8","Type":"ContainerStarted","Data":"e45c8e22895cad3c7536015a661c7dcbbc9c25a67dc62cd601c34618663afb5c"}
Jul 28 20:49:14.803301 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.803281    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-6bkcp"
Jul 28 20:49:14.812744 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.812711    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-6bkcp"
Jul 28 20:49:14.818561 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.818506    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5974c79874-wcg9q" podStartSLOduration=1.487128729 podStartE2EDuration="14.818489048s" totalImagesPullingTime="13.331360319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:49:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:49:14.816948277 +0000 UTC m=+110.041291265" watchObservedRunningTime="2026-07-28 20:49:14.818489048 +0000 UTC m=+110.042832034"
Jul 28 20:49:14.833995 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.833796    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-6bkcp" podStartSLOduration=1.48300523 podStartE2EDuration="20.833773016s" totalImagesPullingTime="19.350767786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:49:14.83369756 +0000 UTC m=+110.058040545" watchObservedRunningTime="2026-07-28 20:49:14.833773016 +0000 UTC m=+110.058116002"
Jul 28 20:49:14.946246 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.946192    2565 patch_prober.go:28] interesting pod/image-registry-586cdbddd-fbphg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 28 20:49:14.946542 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.946510    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-586cdbddd-fbphg" podUID="1ff0086b-af29-49c2-9569-ee3b382929b3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 20:49:14.974959 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.974920    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-64d969f794-5f2bn"]
Jul 28 20:49:14.979973 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.979149    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:14.981753 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.981726    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 20:49:14.982113 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.981838    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 20:49:14.982113 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.982102    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 20:49:14.982277 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.982188    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ntnt8\""
Jul 28 20:49:14.983882 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.983837    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-eobimn5n57n5o\""
Jul 28 20:49:14.984319 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.984187    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 20:49:14.986768 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:14.984802    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 20:49:15.023886 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.023844    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-64d969f794-5f2bn"]
Jul 28 20:49:15.093201 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.093152    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.093402 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.093228    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk77l\" (UniqueName: \"kubernetes.io/projected/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-kube-api-access-wk77l\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.093402 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.093262    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-secret-thanos-querier-tls\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.093402 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.093295    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.093402 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.093337    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.093402 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.093360    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-secret-grpc-tls\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.093402 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.093383    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.093718 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.093508    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-metrics-client-ca\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.194812 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.194020    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-metrics-client-ca\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.194812 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.194115    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.194812 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.194161    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk77l\" (UniqueName: \"kubernetes.io/projected/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-kube-api-access-wk77l\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.194812 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.194194    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-secret-thanos-querier-tls\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.194812 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.194227    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.194812 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.194263    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.194812 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.194290    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-secret-grpc-tls\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.194812 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.194319    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.196057 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.196019    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-metrics-client-ca\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.200872 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.200772    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-secret-grpc-tls\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.204234 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.204066    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.204711 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.204683    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-secret-thanos-querier-tls\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.206355 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.206307    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.206468 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.206431    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.206796 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.206749    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.209175 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.209130    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk77l\" (UniqueName: \"kubernetes.io/projected/ec76b3bd-91e4-4dcd-a677-5abf20db6ed3-kube-api-access-wk77l\") pod \"thanos-querier-64d969f794-5f2bn\" (UID: \"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3\") " pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.297087 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.296809    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:15.557604 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.557525    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-64d969f794-5f2bn"]
Jul 28 20:49:15.827076 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.825675    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5df456bfc9-bgdn5"]
Jul 28 20:49:15.866455 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.866420    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5df456bfc9-bgdn5"]
Jul 28 20:49:15.866677 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.866618    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:15.902294 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.902200    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14a4ec0a-5a0d-4711-8d14-1d526e67c756-console-oauth-config\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:15.902294 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.902285    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14a4ec0a-5a0d-4711-8d14-1d526e67c756-console-serving-cert\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:15.902773 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.902330    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-oauth-serving-cert\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:15.902773 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.902380    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-trusted-ca-bundle\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:15.902773 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.902423    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-console-config\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:15.902773 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.902493    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sphhd\" (UniqueName: \"kubernetes.io/projected/14a4ec0a-5a0d-4711-8d14-1d526e67c756-kube-api-access-sphhd\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:15.902773 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:15.902580    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-service-ca\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:16.003113 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.003072    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-console-config\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:16.003308 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.003146    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sphhd\" (UniqueName: \"kubernetes.io/projected/14a4ec0a-5a0d-4711-8d14-1d526e67c756-kube-api-access-sphhd\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:16.003308 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.003195    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-service-ca\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:16.003308 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.003240    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14a4ec0a-5a0d-4711-8d14-1d526e67c756-console-oauth-config\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:16.003308 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.003275    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14a4ec0a-5a0d-4711-8d14-1d526e67c756-console-serving-cert\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:16.003547 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.003317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-oauth-serving-cert\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:16.003547 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.003367    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-trusted-ca-bundle\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:16.004356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.003928    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-console-config\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:16.004356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.004196    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-trusted-ca-bundle\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:16.005042 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.004951    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-oauth-serving-cert\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:16.005135 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.005108    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-service-ca\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:16.007025 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.007001    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14a4ec0a-5a0d-4711-8d14-1d526e67c756-console-oauth-config\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:16.009336 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.009314    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14a4ec0a-5a0d-4711-8d14-1d526e67c756-console-serving-cert\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:16.012287 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.012225    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sphhd\" (UniqueName: \"kubernetes.io/projected/14a4ec0a-5a0d-4711-8d14-1d526e67c756-kube-api-access-sphhd\") pod \"console-5df456bfc9-bgdn5\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") " pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:16.104932 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:49:16.104827    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec76b3bd_91e4_4dcd_a677_5abf20db6ed3.slice/crio-a35856c9240be0ec115a1b87b80e7cc4efb3559b35abed121c9c0e23b9d63c94 WatchSource:0}: Error finding container a35856c9240be0ec115a1b87b80e7cc4efb3559b35abed121c9c0e23b9d63c94: Status 404 returned error can't find the container with id a35856c9240be0ec115a1b87b80e7cc4efb3559b35abed121c9c0e23b9d63c94
Jul 28 20:49:16.185200 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.184292    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:16.416300 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.416202    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5df456bfc9-bgdn5"]
Jul 28 20:49:16.432409 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:49:16.432330    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a4ec0a_5a0d_4711_8d14_1d526e67c756.slice/crio-b06c223e4c8537eb65f5c2099e4faff94cebfa0a421ebc52938a78ec3cc7f162 WatchSource:0}: Error finding container b06c223e4c8537eb65f5c2099e4faff94cebfa0a421ebc52938a78ec3cc7f162: Status 404 returned error can't find the container with id b06c223e4c8537eb65f5c2099e4faff94cebfa0a421ebc52938a78ec3cc7f162
Jul 28 20:49:16.822697 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.822030    2565 generic.go:356] "Generic (PLEG): container finished" podID="3568e7d8-5fff-4276-83d3-b2d9400d4d40" containerID="e2dc6d5c74b3137639ca0af478743d35d47fc1f4f8827a123ae7f11f00d51a68" exitCode=0
Jul 28 20:49:16.822697 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.822179    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t5fvv" event={"ID":"3568e7d8-5fff-4276-83d3-b2d9400d4d40","Type":"ContainerDied","Data":"e2dc6d5c74b3137639ca0af478743d35d47fc1f4f8827a123ae7f11f00d51a68"}
Jul 28 20:49:16.826603 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.826494    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn" event={"ID":"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3","Type":"ContainerStarted","Data":"a35856c9240be0ec115a1b87b80e7cc4efb3559b35abed121c9c0e23b9d63c94"}
Jul 28 20:49:16.833073 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.832986    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5df456bfc9-bgdn5" event={"ID":"14a4ec0a-5a0d-4711-8d14-1d526e67c756","Type":"ContainerStarted","Data":"260b17cc6499d613bfc381ebacd66b6429818d9896c72c3584e549e283bfbb0e"}
Jul 28 20:49:16.833073 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:16.833035    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5df456bfc9-bgdn5" event={"ID":"14a4ec0a-5a0d-4711-8d14-1d526e67c756","Type":"ContainerStarted","Data":"b06c223e4c8537eb65f5c2099e4faff94cebfa0a421ebc52938a78ec3cc7f162"}
Jul 28 20:49:17.725661 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:17.724957    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-586cdbddd-fbphg"
Jul 28 20:49:17.752042 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:17.751154    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5df456bfc9-bgdn5" podStartSLOduration=2.751127774 podStartE2EDuration="2.751127774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:49:16.888979639 +0000 UTC m=+112.113322705" watchObservedRunningTime="2026-07-28 20:49:17.751127774 +0000 UTC m=+112.975470761"
Jul 28 20:49:17.842145 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:17.842056    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t5fvv" event={"ID":"3568e7d8-5fff-4276-83d3-b2d9400d4d40","Type":"ContainerStarted","Data":"37bcf40590b2a7d34d046dbd3d7594a6d8509bcff90ccbbacdbb82c6ad4754ec"}
Jul 28 20:49:18.850165 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:18.850067    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t5fvv" event={"ID":"3568e7d8-5fff-4276-83d3-b2d9400d4d40","Type":"ContainerStarted","Data":"9b71fa836c5ee462f26fe51ef56b23b6e7c409fd55d3911abc8f7e7e4e9d8013"}
Jul 28 20:49:18.881312 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:18.881240    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t5fvv" podStartSLOduration=5.7059728750000005 podStartE2EDuration="7.881217317s" totalImagesPullingTime="2.045928572s" totalInitContainerRuntime="129.31587ms" isStatefulPod=true podCreationTimestamp="2026-07-28 20:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:49:18.880312417 +0000 UTC m=+114.104655405" watchObservedRunningTime="2026-07-28 20:49:18.881217317 +0000 UTC m=+114.105560305"
Jul 28 20:49:19.576869 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:19.576811    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q" podUID="a635f930-2655-4762-b3f6-67e2f947dc8c" containerName="registry" containerID="cri-o://570b8d8ef08ab9cbd92c78c1a9a3fe65c9c26ae9013940407958bfaf636cf71e" gracePeriod=30
Jul 28 20:49:20.866493 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:20.866454    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:20.866989 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:20.866560    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:20.872914 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:20.872874    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:21.433952 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:21.433871    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5974c79874-wcg9q"]
Jul 28 20:49:21.866807 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:21.866663    2565 generic.go:356] "Generic (PLEG): container finished" podID="a635f930-2655-4762-b3f6-67e2f947dc8c" containerID="570b8d8ef08ab9cbd92c78c1a9a3fe65c9c26ae9013940407958bfaf636cf71e" exitCode=0
Jul 28 20:49:21.867266 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:21.866891    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q" event={"ID":"a635f930-2655-4762-b3f6-67e2f947dc8c","Type":"ContainerDied","Data":"570b8d8ef08ab9cbd92c78c1a9a3fe65c9c26ae9013940407958bfaf636cf71e"}
Jul 28 20:49:21.872356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:21.872330    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:22.494831 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.494798    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:49:22.573108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.572340    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a635f930-2655-4762-b3f6-67e2f947dc8c-ca-trust-extracted\") pod \"a635f930-2655-4762-b3f6-67e2f947dc8c\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") "
Jul 28 20:49:22.573108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.572414    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a635f930-2655-4762-b3f6-67e2f947dc8c-image-registry-private-configuration\") pod \"a635f930-2655-4762-b3f6-67e2f947dc8c\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") "
Jul 28 20:49:22.573108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.572443    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-certificates\") pod \"a635f930-2655-4762-b3f6-67e2f947dc8c\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") "
Jul 28 20:49:22.573108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.572466    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a635f930-2655-4762-b3f6-67e2f947dc8c-installation-pull-secrets\") pod \"a635f930-2655-4762-b3f6-67e2f947dc8c\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") "
Jul 28 20:49:22.573108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.572514    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-bound-sa-token\") pod \"a635f930-2655-4762-b3f6-67e2f947dc8c\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") "
Jul 28 20:49:22.573108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.572575    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a635f930-2655-4762-b3f6-67e2f947dc8c-trusted-ca\") pod \"a635f930-2655-4762-b3f6-67e2f947dc8c\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") "
Jul 28 20:49:22.573108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.572612    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls\") pod \"a635f930-2655-4762-b3f6-67e2f947dc8c\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") "
Jul 28 20:49:22.573108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.572673    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xk9d\" (UniqueName: \"kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-kube-api-access-5xk9d\") pod \"a635f930-2655-4762-b3f6-67e2f947dc8c\" (UID: \"a635f930-2655-4762-b3f6-67e2f947dc8c\") "
Jul 28 20:49:22.573583 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.573234    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a635f930-2655-4762-b3f6-67e2f947dc8c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a635f930-2655-4762-b3f6-67e2f947dc8c" (UID: "a635f930-2655-4762-b3f6-67e2f947dc8c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:49:22.573583 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.573348    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a635f930-2655-4762-b3f6-67e2f947dc8c-trusted-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:22.574313 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.574153    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a635f930-2655-4762-b3f6-67e2f947dc8c" (UID: "a635f930-2655-4762-b3f6-67e2f947dc8c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:49:22.575807 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.575778    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a635f930-2655-4762-b3f6-67e2f947dc8c" (UID: "a635f930-2655-4762-b3f6-67e2f947dc8c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:49:22.576455 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.576414    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a635f930-2655-4762-b3f6-67e2f947dc8c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a635f930-2655-4762-b3f6-67e2f947dc8c" (UID: "a635f930-2655-4762-b3f6-67e2f947dc8c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:49:22.576974 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.576926    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-kube-api-access-5xk9d" (OuterVolumeSpecName: "kube-api-access-5xk9d") pod "a635f930-2655-4762-b3f6-67e2f947dc8c" (UID: "a635f930-2655-4762-b3f6-67e2f947dc8c"). InnerVolumeSpecName "kube-api-access-5xk9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:49:22.578620 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.578590    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a635f930-2655-4762-b3f6-67e2f947dc8c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a635f930-2655-4762-b3f6-67e2f947dc8c" (UID: "a635f930-2655-4762-b3f6-67e2f947dc8c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:49:22.578993 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.578949    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a635f930-2655-4762-b3f6-67e2f947dc8c" (UID: "a635f930-2655-4762-b3f6-67e2f947dc8c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:49:22.586335 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.586303    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a635f930-2655-4762-b3f6-67e2f947dc8c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a635f930-2655-4762-b3f6-67e2f947dc8c" (UID: "a635f930-2655-4762-b3f6-67e2f947dc8c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:49:22.674456 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.674310    2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a635f930-2655-4762-b3f6-67e2f947dc8c-ca-trust-extracted\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:22.674456 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.674339    2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a635f930-2655-4762-b3f6-67e2f947dc8c-image-registry-private-configuration\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:22.674456 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.674354    2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-certificates\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:22.674456 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.674369    2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a635f930-2655-4762-b3f6-67e2f947dc8c-installation-pull-secrets\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:22.674456 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.674383    2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-bound-sa-token\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:22.674456 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.674396    2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-registry-tls\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:22.674456 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.674409    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xk9d\" (UniqueName: \"kubernetes.io/projected/a635f930-2655-4762-b3f6-67e2f947dc8c-kube-api-access-5xk9d\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:22.873699 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.873658    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb" event={"ID":"6525614c-96ff-4222-83eb-a492804d204a","Type":"ContainerStarted","Data":"9d58053a669970047fea2a3d202050ed94db725f9e3bec49dfb4be10bb31b408"}
Jul 28 20:49:22.875358 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.875327    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn" event={"ID":"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3","Type":"ContainerStarted","Data":"24214c164a8e6bd972ce6132042542043cc00c62d43c70c57768a079a5f4bc91"}
Jul 28 20:49:22.877460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.877430    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x6qn9" event={"ID":"41a72d4d-a128-4ed4-934f-0aebbe9825b6","Type":"ContainerStarted","Data":"6b6e3132c23d1c846f3bef50b96d98f08eaccc4392217f002e32b1abfa63e53e"}
Jul 28 20:49:22.878965 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.878832    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q" event={"ID":"a635f930-2655-4762-b3f6-67e2f947dc8c","Type":"ContainerDied","Data":"9719abfa6ca72e65b536d45de30497c4c01fc07885aff324667e23b4f63244d2"}
Jul 28 20:49:22.878965 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.878849    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bb96c4ff7-hbx9q"
Jul 28 20:49:22.878965 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.878881    2565 scope.go:117] "RemoveContainer" containerID="570b8d8ef08ab9cbd92c78c1a9a3fe65c9c26ae9013940407958bfaf636cf71e"
Jul 28 20:49:22.914822 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.914758    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bb96c4ff7-hbx9q"]
Jul 28 20:49:22.922411 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.921280    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-bb96c4ff7-hbx9q"]
Jul 28 20:49:22.959788 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:22.959748    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-42cwb" podStartSLOduration=5.220744448 podStartE2EDuration="12.959732517s" totalImagesPullingTime="7.738988069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:49:22.958281464 +0000 UTC m=+118.182624450" watchObservedRunningTime="2026-07-28 20:49:22.959732517 +0000 UTC m=+118.184075502"
Jul 28 20:49:23.330455 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:23.330412    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a635f930-2655-4762-b3f6-67e2f947dc8c" path="/var/lib/kubelet/pods/a635f930-2655-4762-b3f6-67e2f947dc8c/volumes"
Jul 28 20:49:23.886602 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:23.886514    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn" event={"ID":"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3","Type":"ContainerStarted","Data":"d2e7a1abed565cd66cc8280f52090fc49a01e37706c346bbfc71a0f8ccbb0d01"}
Jul 28 20:49:23.887193 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:23.886606    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn" event={"ID":"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3","Type":"ContainerStarted","Data":"46906e067ccefde350f09134695fbf2cd87826c2b6c28eb17036e36a79b9bf58"}
Jul 28 20:49:23.888731 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:23.888704    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x6qn9" event={"ID":"41a72d4d-a128-4ed4-934f-0aebbe9825b6","Type":"ContainerStarted","Data":"2323c039a135765ebcdf53c359b01d961119bc567e92f1c28dd8f4ec394d6b0e"}
Jul 28 20:49:23.888859 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:23.888805    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:49:23.892244 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:23.892135    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7vtw8" event={"ID":"100dd861-fcc1-45da-ac5b-0797754c195e","Type":"ContainerStarted","Data":"e1d88c618afbd1c0a1c93660905b4e110bad8de0b121baaabd2c7c63433ddf96"}
Jul 28 20:49:23.909387 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:23.909332    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-x6qn9" podStartSLOduration=77.001818371 podStartE2EDuration="1m24.909311089s" totalImagesPullingTime="7.907492718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:49:23.909110954 +0000 UTC m=+119.133453941" watchObservedRunningTime="2026-07-28 20:49:23.909311089 +0000 UTC m=+119.133654075"
Jul 28 20:49:24.899686 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:24.899627    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn" event={"ID":"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3","Type":"ContainerStarted","Data":"0a089c8c614a3e0234fad5a9884dfaddbfea86cf6dba1190bd754efa47b7470f"}
Jul 28 20:49:24.900392 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:24.899738    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn" event={"ID":"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3","Type":"ContainerStarted","Data":"6d3e90ce4cedee889a97a84c8e5e1f4828076e3f4b35f7490720b3511e4268de"}
Jul 28 20:49:24.900392 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:24.899786    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn" event={"ID":"ec76b3bd-91e4-4dcd-a677-5abf20db6ed3","Type":"ContainerStarted","Data":"0f63439bd74ededc17422e1a7236a8df03b7b6992ea5817c710222baf42cb0c8"}
Jul 28 20:49:24.923934 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:24.923885    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn" podStartSLOduration=3.266758383 podStartE2EDuration="10.923866821s" totalImagesPullingTime="7.657108438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:49:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:49:24.923077122 +0000 UTC m=+120.147420110" watchObservedRunningTime="2026-07-28 20:49:24.923866821 +0000 UTC m=+120.148209806"
Jul 28 20:49:25.904206 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:25.904166    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:26.184510 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:26.184473    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:26.184736 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:26.184548    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:26.190036 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:26.190008    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:26.911379 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:26.911351    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:27.913731 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:27.913692    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7vtw8" event={"ID":"100dd861-fcc1-45da-ac5b-0797754c195e","Type":"ContainerStarted","Data":"f2e7ad186f629a652346bf95b55c55c3b2a4e0b505af4980e8e60faf7a69a033"}
Jul 28 20:49:27.936578 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:27.936520    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7vtw8" podStartSLOduration=34.645944418 podStartE2EDuration="1m2.936503227s" totalImagesPullingTime="28.290558809s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:48:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:49:27.934724954 +0000 UTC m=+123.159067951" watchObservedRunningTime="2026-07-28 20:49:27.936503227 +0000 UTC m=+123.160846211"
Jul 28 20:49:31.913774 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:31.913745    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-64d969f794-5f2bn"
Jul 28 20:49:32.158540 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:32.158504    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5df456bfc9-bgdn5"]
Jul 28 20:49:33.902368 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:33.902337    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x6qn9"
Jul 28 20:49:35.192623 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:35.192576    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs\") pod \"network-metrics-daemon-jtxnw\" (UID: \"6c1be7ce-d683-4505-b781-5f739d0d82b5\") " pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:49:35.194994 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:35.194965    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1be7ce-d683-4505-b781-5f739d0d82b5-metrics-certs\") pod \"network-metrics-daemon-jtxnw\" (UID: \"6c1be7ce-d683-4505-b781-5f739d0d82b5\") " pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:49:35.254468 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:35.254432    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gvjsj\""
Jul 28 20:49:35.262260 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:35.262231    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jtxnw"
Jul 28 20:49:35.427259 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:35.427226    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jtxnw"]
Jul 28 20:49:35.430783 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:49:35.430752    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1be7ce_d683_4505_b781_5f739d0d82b5.slice/crio-4da0d2e1e1c1efba1f7d72688e60a7c60e99a683ab163b71dd572f6d5f88ae33 WatchSource:0}: Error finding container 4da0d2e1e1c1efba1f7d72688e60a7c60e99a683ab163b71dd572f6d5f88ae33: Status 404 returned error can't find the container with id 4da0d2e1e1c1efba1f7d72688e60a7c60e99a683ab163b71dd572f6d5f88ae33
Jul 28 20:49:35.948379 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:35.948338    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jtxnw" event={"ID":"6c1be7ce-d683-4505-b781-5f739d0d82b5","Type":"ContainerStarted","Data":"4da0d2e1e1c1efba1f7d72688e60a7c60e99a683ab163b71dd572f6d5f88ae33"}
Jul 28 20:49:37.957013 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:37.956971    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jtxnw" event={"ID":"6c1be7ce-d683-4505-b781-5f739d0d82b5","Type":"ContainerStarted","Data":"71f9ab990557a88dd7c9004a4782170bbc068de5d4209ef3e7a43fbbdbc98b23"}
Jul 28 20:49:37.957573 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:37.957029    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jtxnw" event={"ID":"6c1be7ce-d683-4505-b781-5f739d0d82b5","Type":"ContainerStarted","Data":"a948cd7404be189e4f6aa56927dd3543ebfd68d4b113eb6819c33cd44a54067c"}
Jul 28 20:49:37.976223 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:37.976167    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jtxnw" podStartSLOduration=131.558674377 podStartE2EDuration="2m12.976151707s" totalImagesPullingTime="1.41747733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:49:37.974905551 +0000 UTC m=+133.199248527" watchObservedRunningTime="2026-07-28 20:49:37.976151707 +0000 UTC m=+133.200494693"
Jul 28 20:49:47.925750 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:47.925678    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5974c79874-wcg9q" podUID="7ecc2a5f-ae06-4925-bc69-775b0f422a7f" containerName="console" containerID="cri-o://b4089e44b6d5eaa5b47fe5af74a1e506ee8c906afff24e891242c0de4f74bfb0" gracePeriod=15
Jul 28 20:49:48.188102 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.188076    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:48.302113 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.302079    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-trusted-ca-bundle\") pod \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") "
Jul 28 20:49:48.302303 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.302125    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-console-serving-cert\") pod \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") "
Jul 28 20:49:48.302303 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.302153    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-oauth-serving-cert\") pod \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") "
Jul 28 20:49:48.302303 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.302182    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-console-oauth-config\") pod \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") "
Jul 28 20:49:48.302303 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.302214    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-service-ca\") pod \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") "
Jul 28 20:49:48.302303 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.302231    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kmsm\" (UniqueName: \"kubernetes.io/projected/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-kube-api-access-5kmsm\") pod \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") "
Jul 28 20:49:48.302303 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.302255    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-console-config\") pod \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\" (UID: \"7ecc2a5f-ae06-4925-bc69-775b0f422a7f\") "
Jul 28 20:49:48.302692 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.302661    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7ecc2a5f-ae06-4925-bc69-775b0f422a7f" (UID: "7ecc2a5f-ae06-4925-bc69-775b0f422a7f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:49:48.302776 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.302747    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7ecc2a5f-ae06-4925-bc69-775b0f422a7f" (UID: "7ecc2a5f-ae06-4925-bc69-775b0f422a7f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:49:48.302827 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.302753    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-service-ca" (OuterVolumeSpecName: "service-ca") pod "7ecc2a5f-ae06-4925-bc69-775b0f422a7f" (UID: "7ecc2a5f-ae06-4925-bc69-775b0f422a7f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:49:48.302909 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.302885    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-console-config" (OuterVolumeSpecName: "console-config") pod "7ecc2a5f-ae06-4925-bc69-775b0f422a7f" (UID: "7ecc2a5f-ae06-4925-bc69-775b0f422a7f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:49:48.304670 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.304623    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7ecc2a5f-ae06-4925-bc69-775b0f422a7f" (UID: "7ecc2a5f-ae06-4925-bc69-775b0f422a7f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:49:48.304771 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.304662    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-kube-api-access-5kmsm" (OuterVolumeSpecName: "kube-api-access-5kmsm") pod "7ecc2a5f-ae06-4925-bc69-775b0f422a7f" (UID: "7ecc2a5f-ae06-4925-bc69-775b0f422a7f"). InnerVolumeSpecName "kube-api-access-5kmsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:49:48.304771 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.304720    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7ecc2a5f-ae06-4925-bc69-775b0f422a7f" (UID: "7ecc2a5f-ae06-4925-bc69-775b0f422a7f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:49:48.403069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.403016    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:48.403069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.403063    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kmsm\" (UniqueName: \"kubernetes.io/projected/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-kube-api-access-5kmsm\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:48.403069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.403072    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-console-config\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:48.403069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.403082    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-trusted-ca-bundle\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:48.403069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.403090    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-console-serving-cert\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:48.403369 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.403100    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-oauth-serving-cert\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:48.403369 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:48.403110    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ecc2a5f-ae06-4925-bc69-775b0f422a7f-console-oauth-config\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:49.004238 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:49.004202    2565 generic.go:356] "Generic (PLEG): container finished" podID="7ecc2a5f-ae06-4925-bc69-775b0f422a7f" containerID="b4089e44b6d5eaa5b47fe5af74a1e506ee8c906afff24e891242c0de4f74bfb0" exitCode=0
Jul 28 20:49:49.004238 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:49.004245    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5974c79874-wcg9q" event={"ID":"7ecc2a5f-ae06-4925-bc69-775b0f422a7f","Type":"ContainerDied","Data":"b4089e44b6d5eaa5b47fe5af74a1e506ee8c906afff24e891242c0de4f74bfb0"}
Jul 28 20:49:49.004813 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:49.004259    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5974c79874-wcg9q"
Jul 28 20:49:49.004813 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:49.004285    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5974c79874-wcg9q" event={"ID":"7ecc2a5f-ae06-4925-bc69-775b0f422a7f","Type":"ContainerDied","Data":"cd826be368c6d3dadc258cac8f0ce5e3fc3d3eacfe22bea3e07745db76b13e84"}
Jul 28 20:49:49.004813 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:49.004306    2565 scope.go:117] "RemoveContainer" containerID="b4089e44b6d5eaa5b47fe5af74a1e506ee8c906afff24e891242c0de4f74bfb0"
Jul 28 20:49:49.012304 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:49.012280    2565 scope.go:117] "RemoveContainer" containerID="b4089e44b6d5eaa5b47fe5af74a1e506ee8c906afff24e891242c0de4f74bfb0"
Jul 28 20:49:49.012584 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:49:49.012559    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4089e44b6d5eaa5b47fe5af74a1e506ee8c906afff24e891242c0de4f74bfb0\": container with ID starting with b4089e44b6d5eaa5b47fe5af74a1e506ee8c906afff24e891242c0de4f74bfb0 not found: ID does not exist" containerID="b4089e44b6d5eaa5b47fe5af74a1e506ee8c906afff24e891242c0de4f74bfb0"
Jul 28 20:49:49.012653 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:49.012597    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4089e44b6d5eaa5b47fe5af74a1e506ee8c906afff24e891242c0de4f74bfb0"} err="failed to get container status \"b4089e44b6d5eaa5b47fe5af74a1e506ee8c906afff24e891242c0de4f74bfb0\": rpc error: code = NotFound desc = could not find container \"b4089e44b6d5eaa5b47fe5af74a1e506ee8c906afff24e891242c0de4f74bfb0\": container with ID starting with b4089e44b6d5eaa5b47fe5af74a1e506ee8c906afff24e891242c0de4f74bfb0 not found: ID does not exist"
Jul 28 20:49:49.025629 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:49.025599    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5974c79874-wcg9q"]
Jul 28 20:49:49.026896 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:49.026869    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5974c79874-wcg9q"]
Jul 28 20:49:49.330801 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:49.330709    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ecc2a5f-ae06-4925-bc69-775b0f422a7f" path="/var/lib/kubelet/pods/7ecc2a5f-ae06-4925-bc69-775b0f422a7f/volumes"
Jul 28 20:49:57.179163 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:57.179095    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5df456bfc9-bgdn5" podUID="14a4ec0a-5a0d-4711-8d14-1d526e67c756" containerName="console" containerID="cri-o://260b17cc6499d613bfc381ebacd66b6429818d9896c72c3584e549e283bfbb0e" gracePeriod=15
Jul 28 20:49:57.969920 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:57.969881    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:57.972608 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:57.972589    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14a4ec0a-5a0d-4711-8d14-1d526e67c756-console-serving-cert\") pod \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") "
Jul 28 20:49:57.972724 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:57.972647    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-console-config\") pod \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") "
Jul 28 20:49:57.972724 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:57.972669    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-service-ca\") pod \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") "
Jul 28 20:49:57.972724 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:57.972691    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14a4ec0a-5a0d-4711-8d14-1d526e67c756-console-oauth-config\") pod \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") "
Jul 28 20:49:57.972917 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:57.972886    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-trusted-ca-bundle\") pod \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") "
Jul 28 20:49:57.973055 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:57.972994    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sphhd\" (UniqueName: \"kubernetes.io/projected/14a4ec0a-5a0d-4711-8d14-1d526e67c756-kube-api-access-sphhd\") pod \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") "
Jul 28 20:49:57.973055 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:57.973023    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-oauth-serving-cert\") pod \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\" (UID: \"14a4ec0a-5a0d-4711-8d14-1d526e67c756\") "
Jul 28 20:49:57.973179 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:57.973060    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-service-ca" (OuterVolumeSpecName: "service-ca") pod "14a4ec0a-5a0d-4711-8d14-1d526e67c756" (UID: "14a4ec0a-5a0d-4711-8d14-1d526e67c756"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:49:57.973179 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:57.973111    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-console-config" (OuterVolumeSpecName: "console-config") pod "14a4ec0a-5a0d-4711-8d14-1d526e67c756" (UID: "14a4ec0a-5a0d-4711-8d14-1d526e67c756"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:49:57.973401 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:57.973337    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-console-config\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:57.973401 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:57.973360    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:57.973401 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:57.973375    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "14a4ec0a-5a0d-4711-8d14-1d526e67c756" (UID: "14a4ec0a-5a0d-4711-8d14-1d526e67c756"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:49:57.973574 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:57.973427    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "14a4ec0a-5a0d-4711-8d14-1d526e67c756" (UID: "14a4ec0a-5a0d-4711-8d14-1d526e67c756"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:49:57.975307 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:57.975287    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a4ec0a-5a0d-4711-8d14-1d526e67c756-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "14a4ec0a-5a0d-4711-8d14-1d526e67c756" (UID: "14a4ec0a-5a0d-4711-8d14-1d526e67c756"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:49:57.975415 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:57.975398    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a4ec0a-5a0d-4711-8d14-1d526e67c756-kube-api-access-sphhd" (OuterVolumeSpecName: "kube-api-access-sphhd") pod "14a4ec0a-5a0d-4711-8d14-1d526e67c756" (UID: "14a4ec0a-5a0d-4711-8d14-1d526e67c756"). InnerVolumeSpecName "kube-api-access-sphhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:49:57.975536 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:57.975516    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a4ec0a-5a0d-4711-8d14-1d526e67c756-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "14a4ec0a-5a0d-4711-8d14-1d526e67c756" (UID: "14a4ec0a-5a0d-4711-8d14-1d526e67c756"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:49:58.039783 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:58.039746    2565 generic.go:356] "Generic (PLEG): container finished" podID="14a4ec0a-5a0d-4711-8d14-1d526e67c756" containerID="260b17cc6499d613bfc381ebacd66b6429818d9896c72c3584e549e283bfbb0e" exitCode=0
Jul 28 20:49:58.039960 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:58.039793    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5df456bfc9-bgdn5"
Jul 28 20:49:58.039960 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:58.039791    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5df456bfc9-bgdn5" event={"ID":"14a4ec0a-5a0d-4711-8d14-1d526e67c756","Type":"ContainerDied","Data":"260b17cc6499d613bfc381ebacd66b6429818d9896c72c3584e549e283bfbb0e"}
Jul 28 20:49:58.039960 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:58.039929    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5df456bfc9-bgdn5" event={"ID":"14a4ec0a-5a0d-4711-8d14-1d526e67c756","Type":"ContainerDied","Data":"b06c223e4c8537eb65f5c2099e4faff94cebfa0a421ebc52938a78ec3cc7f162"}
Jul 28 20:49:58.039960 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:58.039950    2565 scope.go:117] "RemoveContainer" containerID="260b17cc6499d613bfc381ebacd66b6429818d9896c72c3584e549e283bfbb0e"
Jul 28 20:49:58.048691 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:58.048673    2565 scope.go:117] "RemoveContainer" containerID="260b17cc6499d613bfc381ebacd66b6429818d9896c72c3584e549e283bfbb0e"
Jul 28 20:49:58.048963 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:49:58.048941    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"260b17cc6499d613bfc381ebacd66b6429818d9896c72c3584e549e283bfbb0e\": container with ID starting with 260b17cc6499d613bfc381ebacd66b6429818d9896c72c3584e549e283bfbb0e not found: ID does not exist" containerID="260b17cc6499d613bfc381ebacd66b6429818d9896c72c3584e549e283bfbb0e"
Jul 28 20:49:58.049014 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:58.048970    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260b17cc6499d613bfc381ebacd66b6429818d9896c72c3584e549e283bfbb0e"} err="failed to get container status \"260b17cc6499d613bfc381ebacd66b6429818d9896c72c3584e549e283bfbb0e\": rpc error: code = NotFound desc = could not find container \"260b17cc6499d613bfc381ebacd66b6429818d9896c72c3584e549e283bfbb0e\": container with ID starting with 260b17cc6499d613bfc381ebacd66b6429818d9896c72c3584e549e283bfbb0e not found: ID does not exist"
Jul 28 20:49:58.061528 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:58.061502    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5df456bfc9-bgdn5"]
Jul 28 20:49:58.066556 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:58.066533    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5df456bfc9-bgdn5"]
Jul 28 20:49:58.074603 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:58.074546    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14a4ec0a-5a0d-4711-8d14-1d526e67c756-console-oauth-config\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:58.074603 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:58.074573    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-trusted-ca-bundle\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:58.074603 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:58.074584    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sphhd\" (UniqueName: \"kubernetes.io/projected/14a4ec0a-5a0d-4711-8d14-1d526e67c756-kube-api-access-sphhd\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:58.074603 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:58.074594    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14a4ec0a-5a0d-4711-8d14-1d526e67c756-oauth-serving-cert\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:58.074603 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:58.074602    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14a4ec0a-5a0d-4711-8d14-1d526e67c756-console-serving-cert\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:49:59.330489 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:49:59.330453    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14a4ec0a-5a0d-4711-8d14-1d526e67c756" path="/var/lib/kubelet/pods/14a4ec0a-5a0d-4711-8d14-1d526e67c756/volumes"
Jul 28 20:50:38.565020 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.564983    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78659d4698-4w96c"]
Jul 28 20:50:38.565496 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.565261    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14a4ec0a-5a0d-4711-8d14-1d526e67c756" containerName="console"
Jul 28 20:50:38.565496 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.565272    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="14a4ec0a-5a0d-4711-8d14-1d526e67c756" containerName="console"
Jul 28 20:50:38.565496 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.565285    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a635f930-2655-4762-b3f6-67e2f947dc8c" containerName="registry"
Jul 28 20:50:38.565496 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.565290    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a635f930-2655-4762-b3f6-67e2f947dc8c" containerName="registry"
Jul 28 20:50:38.565496 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.565303    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ecc2a5f-ae06-4925-bc69-775b0f422a7f" containerName="console"
Jul 28 20:50:38.565496 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.565309    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ecc2a5f-ae06-4925-bc69-775b0f422a7f" containerName="console"
Jul 28 20:50:38.565496 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.565348    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ecc2a5f-ae06-4925-bc69-775b0f422a7f" containerName="console"
Jul 28 20:50:38.565496 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.565356    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a635f930-2655-4762-b3f6-67e2f947dc8c" containerName="registry"
Jul 28 20:50:38.565496 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.565363    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="14a4ec0a-5a0d-4711-8d14-1d526e67c756" containerName="console"
Jul 28 20:50:38.569481 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.569459    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.571318 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.571296    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 20:50:38.571514 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.571467    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 20:50:38.571680 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.571667    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wj57k\""
Jul 28 20:50:38.571970 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.571948    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 20:50:38.572098 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.571995    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 20:50:38.572098 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.572040    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 20:50:38.577373 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.577351    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 20:50:38.581114 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.581090    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78659d4698-4w96c"]
Jul 28 20:50:38.688013 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.687973    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68bfe83f-1cf6-460f-9660-c170a802e1c2-console-serving-cert\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.688192 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.688051    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-trusted-ca-bundle\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.688192 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.688080    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-console-config\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.688192 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.688113    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-service-ca\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.688192 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.688162    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9pld\" (UniqueName: \"kubernetes.io/projected/68bfe83f-1cf6-460f-9660-c170a802e1c2-kube-api-access-c9pld\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.688362 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.688203    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-oauth-serving-cert\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.688362 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.688219    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68bfe83f-1cf6-460f-9660-c170a802e1c2-console-oauth-config\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.788941 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.788904    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-service-ca\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.788941 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.788945    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9pld\" (UniqueName: \"kubernetes.io/projected/68bfe83f-1cf6-460f-9660-c170a802e1c2-kube-api-access-c9pld\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.789198 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.788975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-oauth-serving-cert\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.789198 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.789005    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68bfe83f-1cf6-460f-9660-c170a802e1c2-console-oauth-config\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.789198 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.789034    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68bfe83f-1cf6-460f-9660-c170a802e1c2-console-serving-cert\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.789198 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.789058    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-trusted-ca-bundle\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.789198 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.789084    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-console-config\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.789749 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.789719    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-console-config\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.789883 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.789758    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-service-ca\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.789883 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.789870    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-oauth-serving-cert\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.789999 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.789909    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-trusted-ca-bundle\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.791528 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.791504    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68bfe83f-1cf6-460f-9660-c170a802e1c2-console-serving-cert\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.791618 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.791604    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68bfe83f-1cf6-460f-9660-c170a802e1c2-console-oauth-config\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.797510 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.797484    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9pld\" (UniqueName: \"kubernetes.io/projected/68bfe83f-1cf6-460f-9660-c170a802e1c2-kube-api-access-c9pld\") pod \"console-78659d4698-4w96c\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") " pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:38.880089 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:38.879997    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:39.010476 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:39.010442    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78659d4698-4w96c"]
Jul 28 20:50:39.013579 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:50:39.013551    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68bfe83f_1cf6_460f_9660_c170a802e1c2.slice/crio-b5a0f1838914e7c418dd5e59402a682e6fa9e8df77d8fdedbdfe2928f8b69465 WatchSource:0}: Error finding container b5a0f1838914e7c418dd5e59402a682e6fa9e8df77d8fdedbdfe2928f8b69465: Status 404 returned error can't find the container with id b5a0f1838914e7c418dd5e59402a682e6fa9e8df77d8fdedbdfe2928f8b69465
Jul 28 20:50:39.190963 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:39.190930    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78659d4698-4w96c" event={"ID":"68bfe83f-1cf6-460f-9660-c170a802e1c2","Type":"ContainerStarted","Data":"3ebeedd8c1096cee7fb48b3237686d7b25b0506abce2a450b27bf63cd7c9ac86"}
Jul 28 20:50:39.191125 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:39.191013    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78659d4698-4w96c" event={"ID":"68bfe83f-1cf6-460f-9660-c170a802e1c2","Type":"ContainerStarted","Data":"b5a0f1838914e7c418dd5e59402a682e6fa9e8df77d8fdedbdfe2928f8b69465"}
Jul 28 20:50:39.209540 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:39.209492    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78659d4698-4w96c" podStartSLOduration=1.209478115 podStartE2EDuration="1.209478115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:50:39.208584778 +0000 UTC m=+194.432927764" watchObservedRunningTime="2026-07-28 20:50:39.209478115 +0000 UTC m=+194.433821158"
Jul 28 20:50:48.880391 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:48.880352    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:48.880983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:48.880424    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:48.885853 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:48.885826    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:50:49.231282 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:50:49.231255    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:51:07.655713 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:07.655681    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m"]
Jul 28 20:51:07.659059 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:07.659029    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m"
Jul 28 20:51:07.661123 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:07.661099    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 20:51:07.661624 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:07.661608    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jj4bt\""
Jul 28 20:51:07.661691 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:07.661624    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 20:51:07.668269 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:07.668243    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m"]
Jul 28 20:51:07.823207 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:07.823170    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7060cd33-bac8-49dc-9d28-6b190c3f79a4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m\" (UID: \"7060cd33-bac8-49dc-9d28-6b190c3f79a4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m"
Jul 28 20:51:07.823207 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:07.823213    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfdxb\" (UniqueName: \"kubernetes.io/projected/7060cd33-bac8-49dc-9d28-6b190c3f79a4-kube-api-access-cfdxb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m\" (UID: \"7060cd33-bac8-49dc-9d28-6b190c3f79a4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m"
Jul 28 20:51:07.823425 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:07.823239    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7060cd33-bac8-49dc-9d28-6b190c3f79a4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m\" (UID: \"7060cd33-bac8-49dc-9d28-6b190c3f79a4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m"
Jul 28 20:51:07.923782 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:07.923716    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7060cd33-bac8-49dc-9d28-6b190c3f79a4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m\" (UID: \"7060cd33-bac8-49dc-9d28-6b190c3f79a4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m"
Jul 28 20:51:07.923782 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:07.923752    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdxb\" (UniqueName: \"kubernetes.io/projected/7060cd33-bac8-49dc-9d28-6b190c3f79a4-kube-api-access-cfdxb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m\" (UID: \"7060cd33-bac8-49dc-9d28-6b190c3f79a4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m"
Jul 28 20:51:07.923782 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:07.923774    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7060cd33-bac8-49dc-9d28-6b190c3f79a4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m\" (UID: \"7060cd33-bac8-49dc-9d28-6b190c3f79a4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m"
Jul 28 20:51:07.924089 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:07.924074    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7060cd33-bac8-49dc-9d28-6b190c3f79a4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m\" (UID: \"7060cd33-bac8-49dc-9d28-6b190c3f79a4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m"
Jul 28 20:51:07.924125 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:07.924112    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7060cd33-bac8-49dc-9d28-6b190c3f79a4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m\" (UID: \"7060cd33-bac8-49dc-9d28-6b190c3f79a4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m"
Jul 28 20:51:07.932056 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:07.932041    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfdxb\" (UniqueName: \"kubernetes.io/projected/7060cd33-bac8-49dc-9d28-6b190c3f79a4-kube-api-access-cfdxb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m\" (UID: \"7060cd33-bac8-49dc-9d28-6b190c3f79a4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m"
Jul 28 20:51:07.970032 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:07.970001    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m"
Jul 28 20:51:08.103458 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:08.103433    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m"]
Jul 28 20:51:08.105804 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:51:08.105778    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7060cd33_bac8_49dc_9d28_6b190c3f79a4.slice/crio-2c535a4543a4a8588dda946fed9f677772250704b6961dd2080cf4efd2cadb1c WatchSource:0}: Error finding container 2c535a4543a4a8588dda946fed9f677772250704b6961dd2080cf4efd2cadb1c: Status 404 returned error can't find the container with id 2c535a4543a4a8588dda946fed9f677772250704b6961dd2080cf4efd2cadb1c
Jul 28 20:51:08.296839 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:08.296800    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m" event={"ID":"7060cd33-bac8-49dc-9d28-6b190c3f79a4","Type":"ContainerStarted","Data":"2c535a4543a4a8588dda946fed9f677772250704b6961dd2080cf4efd2cadb1c"}
Jul 28 20:51:15.324900 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:15.324861    2565 generic.go:356] "Generic (PLEG): container finished" podID="7060cd33-bac8-49dc-9d28-6b190c3f79a4" containerID="b3f468d938a8008375afe83f837cc229b93cc0b507f535d6510fd402af82267e" exitCode=0
Jul 28 20:51:15.325306 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:15.324949    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m" event={"ID":"7060cd33-bac8-49dc-9d28-6b190c3f79a4","Type":"ContainerDied","Data":"b3f468d938a8008375afe83f837cc229b93cc0b507f535d6510fd402af82267e"}
Jul 28 20:51:17.334289 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:17.334256    2565 generic.go:356] "Generic (PLEG): container finished" podID="7060cd33-bac8-49dc-9d28-6b190c3f79a4" containerID="9f9017701b5c5da5abb2235d9f7da05b72e282ca251ab1eefdf354d6bf20776e" exitCode=0
Jul 28 20:51:17.334696 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:17.334338    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m" event={"ID":"7060cd33-bac8-49dc-9d28-6b190c3f79a4","Type":"ContainerDied","Data":"9f9017701b5c5da5abb2235d9f7da05b72e282ca251ab1eefdf354d6bf20776e"}
Jul 28 20:51:24.366285 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:24.366250    2565 generic.go:356] "Generic (PLEG): container finished" podID="7060cd33-bac8-49dc-9d28-6b190c3f79a4" containerID="0309210454bcb0efc947925ea0d2994d758c5b76df2f4c8d758ab87fb94ca353" exitCode=0
Jul 28 20:51:24.366722 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:24.366293    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m" event={"ID":"7060cd33-bac8-49dc-9d28-6b190c3f79a4","Type":"ContainerDied","Data":"0309210454bcb0efc947925ea0d2994d758c5b76df2f4c8d758ab87fb94ca353"}
Jul 28 20:51:25.487653 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:25.487614    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m"
Jul 28 20:51:25.569828 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:25.569804    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7060cd33-bac8-49dc-9d28-6b190c3f79a4-bundle\") pod \"7060cd33-bac8-49dc-9d28-6b190c3f79a4\" (UID: \"7060cd33-bac8-49dc-9d28-6b190c3f79a4\") "
Jul 28 20:51:25.569979 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:25.569849    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfdxb\" (UniqueName: \"kubernetes.io/projected/7060cd33-bac8-49dc-9d28-6b190c3f79a4-kube-api-access-cfdxb\") pod \"7060cd33-bac8-49dc-9d28-6b190c3f79a4\" (UID: \"7060cd33-bac8-49dc-9d28-6b190c3f79a4\") "
Jul 28 20:51:25.569979 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:25.569875    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7060cd33-bac8-49dc-9d28-6b190c3f79a4-util\") pod \"7060cd33-bac8-49dc-9d28-6b190c3f79a4\" (UID: \"7060cd33-bac8-49dc-9d28-6b190c3f79a4\") "
Jul 28 20:51:25.570418 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:25.570389    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7060cd33-bac8-49dc-9d28-6b190c3f79a4-bundle" (OuterVolumeSpecName: "bundle") pod "7060cd33-bac8-49dc-9d28-6b190c3f79a4" (UID: "7060cd33-bac8-49dc-9d28-6b190c3f79a4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:51:25.571885 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:25.571862    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7060cd33-bac8-49dc-9d28-6b190c3f79a4-kube-api-access-cfdxb" (OuterVolumeSpecName: "kube-api-access-cfdxb") pod "7060cd33-bac8-49dc-9d28-6b190c3f79a4" (UID: "7060cd33-bac8-49dc-9d28-6b190c3f79a4"). InnerVolumeSpecName "kube-api-access-cfdxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:51:25.573783 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:25.573763    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7060cd33-bac8-49dc-9d28-6b190c3f79a4-util" (OuterVolumeSpecName: "util") pod "7060cd33-bac8-49dc-9d28-6b190c3f79a4" (UID: "7060cd33-bac8-49dc-9d28-6b190c3f79a4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:51:25.671134 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:25.671042    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfdxb\" (UniqueName: \"kubernetes.io/projected/7060cd33-bac8-49dc-9d28-6b190c3f79a4-kube-api-access-cfdxb\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:51:25.671134 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:25.671071    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7060cd33-bac8-49dc-9d28-6b190c3f79a4-util\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:51:25.671134 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:25.671080    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7060cd33-bac8-49dc-9d28-6b190c3f79a4-bundle\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:51:26.375761 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:26.375728    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m" event={"ID":"7060cd33-bac8-49dc-9d28-6b190c3f79a4","Type":"ContainerDied","Data":"2c535a4543a4a8588dda946fed9f677772250704b6961dd2080cf4efd2cadb1c"}
Jul 28 20:51:26.375761 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:26.375753    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dwv5m"
Jul 28 20:51:26.375972 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:26.375773    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c535a4543a4a8588dda946fed9f677772250704b6961dd2080cf4efd2cadb1c"
Jul 28 20:51:30.669922 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:30.669886    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hs68k"]
Jul 28 20:51:30.670497 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:30.670298    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7060cd33-bac8-49dc-9d28-6b190c3f79a4" containerName="util"
Jul 28 20:51:30.670497 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:30.670316    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7060cd33-bac8-49dc-9d28-6b190c3f79a4" containerName="util"
Jul 28 20:51:30.670497 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:30.670336    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7060cd33-bac8-49dc-9d28-6b190c3f79a4" containerName="pull"
Jul 28 20:51:30.670497 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:30.670344    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7060cd33-bac8-49dc-9d28-6b190c3f79a4" containerName="pull"
Jul 28 20:51:30.670497 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:30.670372    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7060cd33-bac8-49dc-9d28-6b190c3f79a4" containerName="extract"
Jul 28 20:51:30.670497 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:30.670380    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7060cd33-bac8-49dc-9d28-6b190c3f79a4" containerName="extract"
Jul 28 20:51:30.670497 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:30.670446    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7060cd33-bac8-49dc-9d28-6b190c3f79a4" containerName="extract"
Jul 28 20:51:30.714461 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:30.714425    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hs68k"]
Jul 28 20:51:30.714666 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:30.714542    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hs68k"
Jul 28 20:51:30.716690 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:30.716665    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 28 20:51:30.716841 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:30.716759    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 28 20:51:30.716914 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:30.716855    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9476g\""
Jul 28 20:51:30.808480 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:30.808444    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efd0c07a-bdb3-4189-be91-f16495268622-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hs68k\" (UID: \"efd0c07a-bdb3-4189-be91-f16495268622\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hs68k"
Jul 28 20:51:30.808678 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:30.808512    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8djt\" (UniqueName: \"kubernetes.io/projected/efd0c07a-bdb3-4189-be91-f16495268622-kube-api-access-p8djt\") pod \"cert-manager-operator-controller-manager-55c444bf86-hs68k\" (UID: \"efd0c07a-bdb3-4189-be91-f16495268622\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hs68k"
Jul 28 20:51:30.909035 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:30.909007    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8djt\" (UniqueName: \"kubernetes.io/projected/efd0c07a-bdb3-4189-be91-f16495268622-kube-api-access-p8djt\") pod \"cert-manager-operator-controller-manager-55c444bf86-hs68k\" (UID: \"efd0c07a-bdb3-4189-be91-f16495268622\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hs68k"
Jul 28 20:51:30.909186 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:30.909052    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efd0c07a-bdb3-4189-be91-f16495268622-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hs68k\" (UID: \"efd0c07a-bdb3-4189-be91-f16495268622\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hs68k"
Jul 28 20:51:30.909379 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:30.909365    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efd0c07a-bdb3-4189-be91-f16495268622-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hs68k\" (UID: \"efd0c07a-bdb3-4189-be91-f16495268622\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hs68k"
Jul 28 20:51:30.917666 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:30.917620    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8djt\" (UniqueName: \"kubernetes.io/projected/efd0c07a-bdb3-4189-be91-f16495268622-kube-api-access-p8djt\") pod \"cert-manager-operator-controller-manager-55c444bf86-hs68k\" (UID: \"efd0c07a-bdb3-4189-be91-f16495268622\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hs68k"
Jul 28 20:51:31.023829 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:31.023797    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hs68k"
Jul 28 20:51:31.188348 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:31.188318    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hs68k"]
Jul 28 20:51:31.191990 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:51:31.191959    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd0c07a_bdb3_4189_be91_f16495268622.slice/crio-b4124b18c20ea53144c30bf411207ccfd20bd25165ae921a39c433dbae80d178 WatchSource:0}: Error finding container b4124b18c20ea53144c30bf411207ccfd20bd25165ae921a39c433dbae80d178: Status 404 returned error can't find the container with id b4124b18c20ea53144c30bf411207ccfd20bd25165ae921a39c433dbae80d178
Jul 28 20:51:31.394556 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:31.394481    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hs68k" event={"ID":"efd0c07a-bdb3-4189-be91-f16495268622","Type":"ContainerStarted","Data":"b4124b18c20ea53144c30bf411207ccfd20bd25165ae921a39c433dbae80d178"}
Jul 28 20:51:33.404513 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:33.404409    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hs68k" event={"ID":"efd0c07a-bdb3-4189-be91-f16495268622","Type":"ContainerStarted","Data":"91413dc8a5e761f1046ce247ecd3dfc8f809d04eb3e87fb0e9a113fa834bf92d"}
Jul 28 20:51:33.424462 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:33.424401    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hs68k" podStartSLOduration=1.503323934 podStartE2EDuration="3.424371921s" totalImagesPullingTime="1.921047987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:51:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:51:33.42295064 +0000 UTC m=+248.647293625" watchObservedRunningTime="2026-07-28 20:51:33.424371921 +0000 UTC m=+248.648714907"
Jul 28 20:51:34.521117 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:34.521085    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md"]
Jul 28 20:51:34.524681 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:34.524663    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md"
Jul 28 20:51:34.526730 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:34.526709    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 20:51:34.526730 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:34.526727    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jj4bt\""
Jul 28 20:51:34.526982 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:34.526966    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 20:51:34.535332 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:34.535311    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md"]
Jul 28 20:51:34.642963 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:34.642931    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a7a4b86-c81a-4807-8591-c958f4bf93e7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md\" (UID: \"2a7a4b86-c81a-4807-8591-c958f4bf93e7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md"
Jul 28 20:51:34.642963 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:34.642975    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whdh4\" (UniqueName: \"kubernetes.io/projected/2a7a4b86-c81a-4807-8591-c958f4bf93e7-kube-api-access-whdh4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md\" (UID: \"2a7a4b86-c81a-4807-8591-c958f4bf93e7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md"
Jul 28 20:51:34.643211 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:34.643002    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a7a4b86-c81a-4807-8591-c958f4bf93e7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md\" (UID: \"2a7a4b86-c81a-4807-8591-c958f4bf93e7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md"
Jul 28 20:51:34.744431 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:34.744383    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whdh4\" (UniqueName: \"kubernetes.io/projected/2a7a4b86-c81a-4807-8591-c958f4bf93e7-kube-api-access-whdh4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md\" (UID: \"2a7a4b86-c81a-4807-8591-c958f4bf93e7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md"
Jul 28 20:51:34.744610 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:34.744440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a7a4b86-c81a-4807-8591-c958f4bf93e7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md\" (UID: \"2a7a4b86-c81a-4807-8591-c958f4bf93e7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md"
Jul 28 20:51:34.744610 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:34.744551    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a7a4b86-c81a-4807-8591-c958f4bf93e7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md\" (UID: \"2a7a4b86-c81a-4807-8591-c958f4bf93e7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md"
Jul 28 20:51:34.744790 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:34.744769    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a7a4b86-c81a-4807-8591-c958f4bf93e7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md\" (UID: \"2a7a4b86-c81a-4807-8591-c958f4bf93e7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md"
Jul 28 20:51:34.744880 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:34.744861    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a7a4b86-c81a-4807-8591-c958f4bf93e7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md\" (UID: \"2a7a4b86-c81a-4807-8591-c958f4bf93e7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md"
Jul 28 20:51:34.752617 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:34.752590    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whdh4\" (UniqueName: \"kubernetes.io/projected/2a7a4b86-c81a-4807-8591-c958f4bf93e7-kube-api-access-whdh4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md\" (UID: \"2a7a4b86-c81a-4807-8591-c958f4bf93e7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md"
Jul 28 20:51:34.834053 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:34.833971    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md"
Jul 28 20:51:34.969111 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:34.969014    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md"]
Jul 28 20:51:34.971625 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:51:34.971597    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7a4b86_c81a_4807_8591_c958f4bf93e7.slice/crio-ec5396fb20ed5886f85f4c92a9c28b1db7d75fe98bf57500e5764bcd9e3e04a6 WatchSource:0}: Error finding container ec5396fb20ed5886f85f4c92a9c28b1db7d75fe98bf57500e5764bcd9e3e04a6: Status 404 returned error can't find the container with id ec5396fb20ed5886f85f4c92a9c28b1db7d75fe98bf57500e5764bcd9e3e04a6
Jul 28 20:51:35.414229 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:35.414197    2565 generic.go:356] "Generic (PLEG): container finished" podID="2a7a4b86-c81a-4807-8591-c958f4bf93e7" containerID="69c41569917a069a18d878f03d0a204fdc83bdf080952ca19baa277721b017e4" exitCode=0
Jul 28 20:51:35.414387 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:35.414284    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md" event={"ID":"2a7a4b86-c81a-4807-8591-c958f4bf93e7","Type":"ContainerDied","Data":"69c41569917a069a18d878f03d0a204fdc83bdf080952ca19baa277721b017e4"}
Jul 28 20:51:35.414387 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:35.414332    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md" event={"ID":"2a7a4b86-c81a-4807-8591-c958f4bf93e7","Type":"ContainerStarted","Data":"ec5396fb20ed5886f85f4c92a9c28b1db7d75fe98bf57500e5764bcd9e3e04a6"}
Jul 28 20:51:37.051798 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:37.051756    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pk26z"]
Jul 28 20:51:37.061226 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:37.061201    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pk26z"
Jul 28 20:51:37.063616 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:37.063585    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 28 20:51:37.064062 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:37.063927    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 28 20:51:37.064432 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:37.064236    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5hm6r\""
Jul 28 20:51:37.066543 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:37.066523    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pk26z"]
Jul 28 20:51:37.164753 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:37.164679    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/369d5fae-78d5-44cb-a325-431521a77eaa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pk26z\" (UID: \"369d5fae-78d5-44cb-a325-431521a77eaa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pk26z"
Jul 28 20:51:37.164937 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:37.164852    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8cl7\" (UniqueName: \"kubernetes.io/projected/369d5fae-78d5-44cb-a325-431521a77eaa-kube-api-access-p8cl7\") pod \"cert-manager-cainjector-559cdd8c87-pk26z\" (UID: \"369d5fae-78d5-44cb-a325-431521a77eaa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pk26z"
Jul 28 20:51:37.265991 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:37.265946    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8cl7\" (UniqueName: \"kubernetes.io/projected/369d5fae-78d5-44cb-a325-431521a77eaa-kube-api-access-p8cl7\") pod \"cert-manager-cainjector-559cdd8c87-pk26z\" (UID: \"369d5fae-78d5-44cb-a325-431521a77eaa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pk26z"
Jul 28 20:51:37.265991 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:37.265998    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/369d5fae-78d5-44cb-a325-431521a77eaa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pk26z\" (UID: \"369d5fae-78d5-44cb-a325-431521a77eaa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pk26z"
Jul 28 20:51:37.275602 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:37.275557    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8cl7\" (UniqueName: \"kubernetes.io/projected/369d5fae-78d5-44cb-a325-431521a77eaa-kube-api-access-p8cl7\") pod \"cert-manager-cainjector-559cdd8c87-pk26z\" (UID: \"369d5fae-78d5-44cb-a325-431521a77eaa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pk26z"
Jul 28 20:51:37.275796 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:37.275766    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/369d5fae-78d5-44cb-a325-431521a77eaa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pk26z\" (UID: \"369d5fae-78d5-44cb-a325-431521a77eaa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pk26z"
Jul 28 20:51:37.380254 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:37.380231    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pk26z"
Jul 28 20:51:37.529294 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:37.529267    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pk26z"]
Jul 28 20:51:37.574516 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:51:37.574483    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod369d5fae_78d5_44cb_a325_431521a77eaa.slice/crio-bd4c47f3cede504b20010af6b7a41db171611641bd949f90d7c728b2a6dde23e WatchSource:0}: Error finding container bd4c47f3cede504b20010af6b7a41db171611641bd949f90d7c728b2a6dde23e: Status 404 returned error can't find the container with id bd4c47f3cede504b20010af6b7a41db171611641bd949f90d7c728b2a6dde23e
Jul 28 20:51:38.430855 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:38.430813    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pk26z" event={"ID":"369d5fae-78d5-44cb-a325-431521a77eaa","Type":"ContainerStarted","Data":"bd4c47f3cede504b20010af6b7a41db171611641bd949f90d7c728b2a6dde23e"}
Jul 28 20:51:38.432748 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:38.432718    2565 generic.go:356] "Generic (PLEG): container finished" podID="2a7a4b86-c81a-4807-8591-c958f4bf93e7" containerID="d4bb8564e3807220d79b4ccfa3ed80e735204699c1de5c701b804ede49c49609" exitCode=0
Jul 28 20:51:38.432873 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:38.432758    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md" event={"ID":"2a7a4b86-c81a-4807-8591-c958f4bf93e7","Type":"ContainerDied","Data":"d4bb8564e3807220d79b4ccfa3ed80e735204699c1de5c701b804ede49c49609"}
Jul 28 20:51:39.440877 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:39.440841    2565 generic.go:356] "Generic (PLEG): container finished" podID="2a7a4b86-c81a-4807-8591-c958f4bf93e7" containerID="99dfc2dd57c0589a2ee786acf5ec8b64769fac6fcd281c1f9b272f2a0a9bf741" exitCode=0
Jul 28 20:51:39.441254 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:39.440919    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md" event={"ID":"2a7a4b86-c81a-4807-8591-c958f4bf93e7","Type":"ContainerDied","Data":"99dfc2dd57c0589a2ee786acf5ec8b64769fac6fcd281c1f9b272f2a0a9bf741"}
Jul 28 20:51:40.447863 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:40.447761    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pk26z" event={"ID":"369d5fae-78d5-44cb-a325-431521a77eaa","Type":"ContainerStarted","Data":"8eca730f4ae9cc171666a962b82c6c77cfdd1d23125eda1859a1452cd2d4e5d1"}
Jul 28 20:51:40.464836 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:40.464784    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pk26z" podStartSLOduration=0.853401586 podStartE2EDuration="3.46476577s" totalImagesPullingTime="2.611364184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:51:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:51:40.464309814 +0000 UTC m=+255.688652802" watchObservedRunningTime="2026-07-28 20:51:40.46476577 +0000 UTC m=+255.689108760"
Jul 28 20:51:40.593206 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:40.593176    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md"
Jul 28 20:51:40.696511 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:40.696467    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whdh4\" (UniqueName: \"kubernetes.io/projected/2a7a4b86-c81a-4807-8591-c958f4bf93e7-kube-api-access-whdh4\") pod \"2a7a4b86-c81a-4807-8591-c958f4bf93e7\" (UID: \"2a7a4b86-c81a-4807-8591-c958f4bf93e7\") "
Jul 28 20:51:40.696704 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:40.696530    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a7a4b86-c81a-4807-8591-c958f4bf93e7-bundle\") pod \"2a7a4b86-c81a-4807-8591-c958f4bf93e7\" (UID: \"2a7a4b86-c81a-4807-8591-c958f4bf93e7\") "
Jul 28 20:51:40.696704 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:40.696550    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a7a4b86-c81a-4807-8591-c958f4bf93e7-util\") pod \"2a7a4b86-c81a-4807-8591-c958f4bf93e7\" (UID: \"2a7a4b86-c81a-4807-8591-c958f4bf93e7\") "
Jul 28 20:51:40.696968 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:40.696946    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7a4b86-c81a-4807-8591-c958f4bf93e7-bundle" (OuterVolumeSpecName: "bundle") pod "2a7a4b86-c81a-4807-8591-c958f4bf93e7" (UID: "2a7a4b86-c81a-4807-8591-c958f4bf93e7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:51:40.698660 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:40.698585    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7a4b86-c81a-4807-8591-c958f4bf93e7-kube-api-access-whdh4" (OuterVolumeSpecName: "kube-api-access-whdh4") pod "2a7a4b86-c81a-4807-8591-c958f4bf93e7" (UID: "2a7a4b86-c81a-4807-8591-c958f4bf93e7"). InnerVolumeSpecName "kube-api-access-whdh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:51:40.700822 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:40.700794    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7a4b86-c81a-4807-8591-c958f4bf93e7-util" (OuterVolumeSpecName: "util") pod "2a7a4b86-c81a-4807-8591-c958f4bf93e7" (UID: "2a7a4b86-c81a-4807-8591-c958f4bf93e7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:51:40.798025 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:40.797989    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a7a4b86-c81a-4807-8591-c958f4bf93e7-bundle\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:51:40.798025 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:40.798018    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a7a4b86-c81a-4807-8591-c958f4bf93e7-util\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:51:40.798025 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:40.798028    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whdh4\" (UniqueName: \"kubernetes.io/projected/2a7a4b86-c81a-4807-8591-c958f4bf93e7-kube-api-access-whdh4\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:51:41.453631 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:41.453595    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md" event={"ID":"2a7a4b86-c81a-4807-8591-c958f4bf93e7","Type":"ContainerDied","Data":"ec5396fb20ed5886f85f4c92a9c28b1db7d75fe98bf57500e5764bcd9e3e04a6"}
Jul 28 20:51:41.453631 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:41.453580    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhj5md"
Jul 28 20:51:41.454162 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:41.453666    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec5396fb20ed5886f85f4c92a9c28b1db7d75fe98bf57500e5764bcd9e3e04a6"
Jul 28 20:51:47.689188 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:47.689106    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gldg9"]
Jul 28 20:51:47.689573 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:47.689406    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a7a4b86-c81a-4807-8591-c958f4bf93e7" containerName="extract"
Jul 28 20:51:47.689573 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:47.689417    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a7a4b86-c81a-4807-8591-c958f4bf93e7" containerName="extract"
Jul 28 20:51:47.689573 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:47.689433    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a7a4b86-c81a-4807-8591-c958f4bf93e7" containerName="util"
Jul 28 20:51:47.689573 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:47.689439    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a7a4b86-c81a-4807-8591-c958f4bf93e7" containerName="util"
Jul 28 20:51:47.689573 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:47.689446    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a7a4b86-c81a-4807-8591-c958f4bf93e7" containerName="pull"
Jul 28 20:51:47.689573 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:47.689451    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a7a4b86-c81a-4807-8591-c958f4bf93e7" containerName="pull"
Jul 28 20:51:47.689573 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:47.689504    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a7a4b86-c81a-4807-8591-c958f4bf93e7" containerName="extract"
Jul 28 20:51:47.692218 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:47.692203    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gldg9"
Jul 28 20:51:47.696610 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:47.696590    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 20:51:47.696737 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:47.696590    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 20:51:47.696737 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:47.696687    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-9vd2p\""
Jul 28 20:51:47.703125 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:47.703106    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gldg9"]
Jul 28 20:51:47.754525 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:47.754491    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02c59153-c394-4b59-adbc-1e6416b3cea2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gldg9\" (UID: \"02c59153-c394-4b59-adbc-1e6416b3cea2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gldg9"
Jul 28 20:51:47.754525 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:47.754527    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvrcn\" (UniqueName: \"kubernetes.io/projected/02c59153-c394-4b59-adbc-1e6416b3cea2-kube-api-access-pvrcn\") pod \"openshift-lws-operator-fd8ccff4c-gldg9\" (UID: \"02c59153-c394-4b59-adbc-1e6416b3cea2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gldg9"
Jul 28 20:51:47.855237 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:47.855197    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvrcn\" (UniqueName: \"kubernetes.io/projected/02c59153-c394-4b59-adbc-1e6416b3cea2-kube-api-access-pvrcn\") pod \"openshift-lws-operator-fd8ccff4c-gldg9\" (UID: \"02c59153-c394-4b59-adbc-1e6416b3cea2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gldg9"
Jul 28 20:51:47.855366 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:47.855289    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02c59153-c394-4b59-adbc-1e6416b3cea2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gldg9\" (UID: \"02c59153-c394-4b59-adbc-1e6416b3cea2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gldg9"
Jul 28 20:51:47.855620 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:47.855603    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02c59153-c394-4b59-adbc-1e6416b3cea2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gldg9\" (UID: \"02c59153-c394-4b59-adbc-1e6416b3cea2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gldg9"
Jul 28 20:51:47.863540 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:47.863511    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvrcn\" (UniqueName: \"kubernetes.io/projected/02c59153-c394-4b59-adbc-1e6416b3cea2-kube-api-access-pvrcn\") pod \"openshift-lws-operator-fd8ccff4c-gldg9\" (UID: \"02c59153-c394-4b59-adbc-1e6416b3cea2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gldg9"
Jul 28 20:51:48.001356 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:48.001280    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gldg9"
Jul 28 20:51:48.129952 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:48.129919    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gldg9"]
Jul 28 20:51:48.133151 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:51:48.133125    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02c59153_c394_4b59_adbc_1e6416b3cea2.slice/crio-8034810ab8136c7d2b915b4773499d8a14624a0a593b8b0c949b65b34e71a6a7 WatchSource:0}: Error finding container 8034810ab8136c7d2b915b4773499d8a14624a0a593b8b0c949b65b34e71a6a7: Status 404 returned error can't find the container with id 8034810ab8136c7d2b915b4773499d8a14624a0a593b8b0c949b65b34e71a6a7
Jul 28 20:51:48.483312 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:48.483272    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gldg9" event={"ID":"02c59153-c394-4b59-adbc-1e6416b3cea2","Type":"ContainerStarted","Data":"8034810ab8136c7d2b915b4773499d8a14624a0a593b8b0c949b65b34e71a6a7"}
Jul 28 20:51:51.497959 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:51.497918    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gldg9" event={"ID":"02c59153-c394-4b59-adbc-1e6416b3cea2","Type":"ContainerStarted","Data":"d0ed91a69f7006187ef04721542558c6f1dd87ee9cb9afcdb6046ba163ec29ec"}
Jul 28 20:51:51.522252 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:51.522194    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gldg9" podStartSLOduration=1.947561184 podStartE2EDuration="4.522172257s" totalImagesPullingTime="2.574611073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:51:51.519700314 +0000 UTC m=+266.744043300" watchObservedRunningTime="2026-07-28 20:51:51.522172257 +0000 UTC m=+266.746515245"
Jul 28 20:51:55.148343 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:55.148304    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw"]
Jul 28 20:51:55.170299 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:55.170275    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw"]
Jul 28 20:51:55.170449 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:55.170370    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw"
Jul 28 20:51:55.172770 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:55.172742    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jj4bt\""
Jul 28 20:51:55.172900 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:55.172765    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 20:51:55.172900 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:55.172790    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 20:51:55.323363 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:55.323331    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73f104ec-3e73-4db9-888c-0e1174583af1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw\" (UID: \"73f104ec-3e73-4db9-888c-0e1174583af1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw"
Jul 28 20:51:55.323363 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:55.323367    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8bwm\" (UniqueName: \"kubernetes.io/projected/73f104ec-3e73-4db9-888c-0e1174583af1-kube-api-access-t8bwm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw\" (UID: \"73f104ec-3e73-4db9-888c-0e1174583af1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw"
Jul 28 20:51:55.323576 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:55.323394    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73f104ec-3e73-4db9-888c-0e1174583af1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw\" (UID: \"73f104ec-3e73-4db9-888c-0e1174583af1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw"
Jul 28 20:51:55.424797 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:55.424704    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73f104ec-3e73-4db9-888c-0e1174583af1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw\" (UID: \"73f104ec-3e73-4db9-888c-0e1174583af1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw"
Jul 28 20:51:55.424797 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:55.424742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8bwm\" (UniqueName: \"kubernetes.io/projected/73f104ec-3e73-4db9-888c-0e1174583af1-kube-api-access-t8bwm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw\" (UID: \"73f104ec-3e73-4db9-888c-0e1174583af1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw"
Jul 28 20:51:55.424797 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:55.424780    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73f104ec-3e73-4db9-888c-0e1174583af1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw\" (UID: \"73f104ec-3e73-4db9-888c-0e1174583af1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw"
Jul 28 20:51:55.425173 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:55.425144    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73f104ec-3e73-4db9-888c-0e1174583af1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw\" (UID: \"73f104ec-3e73-4db9-888c-0e1174583af1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw"
Jul 28 20:51:55.425226 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:55.425190    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73f104ec-3e73-4db9-888c-0e1174583af1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw\" (UID: \"73f104ec-3e73-4db9-888c-0e1174583af1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw"
Jul 28 20:51:55.432887 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:55.432858    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8bwm\" (UniqueName: \"kubernetes.io/projected/73f104ec-3e73-4db9-888c-0e1174583af1-kube-api-access-t8bwm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw\" (UID: \"73f104ec-3e73-4db9-888c-0e1174583af1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw"
Jul 28 20:51:55.479436 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:55.479410    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw"
Jul 28 20:51:55.607803 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:55.607778    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw"]
Jul 28 20:51:55.610222 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:51:55.610198    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f104ec_3e73_4db9_888c_0e1174583af1.slice/crio-46dad53e6dbe1c59029ab6f83b7584462783dd085db80e8d7260c5a0557f5851 WatchSource:0}: Error finding container 46dad53e6dbe1c59029ab6f83b7584462783dd085db80e8d7260c5a0557f5851: Status 404 returned error can't find the container with id 46dad53e6dbe1c59029ab6f83b7584462783dd085db80e8d7260c5a0557f5851
Jul 28 20:51:56.519598 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:56.519562    2565 generic.go:356] "Generic (PLEG): container finished" podID="73f104ec-3e73-4db9-888c-0e1174583af1" containerID="65c1131f1b49330a60caccb964fbe43346998abcc2ce463bfd54d5f54cd7990a" exitCode=0
Jul 28 20:51:56.519989 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:56.519652    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw" event={"ID":"73f104ec-3e73-4db9-888c-0e1174583af1","Type":"ContainerDied","Data":"65c1131f1b49330a60caccb964fbe43346998abcc2ce463bfd54d5f54cd7990a"}
Jul 28 20:51:56.519989 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:56.519721    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw" event={"ID":"73f104ec-3e73-4db9-888c-0e1174583af1","Type":"ContainerStarted","Data":"46dad53e6dbe1c59029ab6f83b7584462783dd085db80e8d7260c5a0557f5851"}
Jul 28 20:51:57.525683 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:57.525650    2565 generic.go:356] "Generic (PLEG): container finished" podID="73f104ec-3e73-4db9-888c-0e1174583af1" containerID="3ecf8993141e6c63b2d3f36917a18d6c7316f45eaded4fb1a549c8d1dabe7157" exitCode=0
Jul 28 20:51:57.526055 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:57.525735    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw" event={"ID":"73f104ec-3e73-4db9-888c-0e1174583af1","Type":"ContainerDied","Data":"3ecf8993141e6c63b2d3f36917a18d6c7316f45eaded4fb1a549c8d1dabe7157"}
Jul 28 20:51:58.532211 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:58.532177    2565 generic.go:356] "Generic (PLEG): container finished" podID="73f104ec-3e73-4db9-888c-0e1174583af1" containerID="8db142f66d95987c6e87fc8a6658f222deb6619ef852505279d510fc664c5dc5" exitCode=0
Jul 28 20:51:58.532618 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:58.532271    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw" event={"ID":"73f104ec-3e73-4db9-888c-0e1174583af1","Type":"ContainerDied","Data":"8db142f66d95987c6e87fc8a6658f222deb6619ef852505279d510fc664c5dc5"}
Jul 28 20:51:59.671432 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:59.671411    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw"
Jul 28 20:51:59.758458 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:59.758422    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73f104ec-3e73-4db9-888c-0e1174583af1-util\") pod \"73f104ec-3e73-4db9-888c-0e1174583af1\" (UID: \"73f104ec-3e73-4db9-888c-0e1174583af1\") "
Jul 28 20:51:59.758674 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:59.758499    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73f104ec-3e73-4db9-888c-0e1174583af1-bundle\") pod \"73f104ec-3e73-4db9-888c-0e1174583af1\" (UID: \"73f104ec-3e73-4db9-888c-0e1174583af1\") "
Jul 28 20:51:59.758674 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:59.758536    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8bwm\" (UniqueName: \"kubernetes.io/projected/73f104ec-3e73-4db9-888c-0e1174583af1-kube-api-access-t8bwm\") pod \"73f104ec-3e73-4db9-888c-0e1174583af1\" (UID: \"73f104ec-3e73-4db9-888c-0e1174583af1\") "
Jul 28 20:51:59.759282 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:59.759253    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f104ec-3e73-4db9-888c-0e1174583af1-bundle" (OuterVolumeSpecName: "bundle") pod "73f104ec-3e73-4db9-888c-0e1174583af1" (UID: "73f104ec-3e73-4db9-888c-0e1174583af1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:51:59.760614 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:59.760588    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f104ec-3e73-4db9-888c-0e1174583af1-kube-api-access-t8bwm" (OuterVolumeSpecName: "kube-api-access-t8bwm") pod "73f104ec-3e73-4db9-888c-0e1174583af1" (UID: "73f104ec-3e73-4db9-888c-0e1174583af1"). InnerVolumeSpecName "kube-api-access-t8bwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:51:59.764146 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:59.764118    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f104ec-3e73-4db9-888c-0e1174583af1-util" (OuterVolumeSpecName: "util") pod "73f104ec-3e73-4db9-888c-0e1174583af1" (UID: "73f104ec-3e73-4db9-888c-0e1174583af1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:51:59.860100 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:59.860015    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73f104ec-3e73-4db9-888c-0e1174583af1-util\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:51:59.860100 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:59.860045    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73f104ec-3e73-4db9-888c-0e1174583af1-bundle\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:51:59.860100 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:51:59.860055    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8bwm\" (UniqueName: \"kubernetes.io/projected/73f104ec-3e73-4db9-888c-0e1174583af1-kube-api-access-t8bwm\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:52:00.542298 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:00.542263    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw" event={"ID":"73f104ec-3e73-4db9-888c-0e1174583af1","Type":"ContainerDied","Data":"46dad53e6dbe1c59029ab6f83b7584462783dd085db80e8d7260c5a0557f5851"}
Jul 28 20:52:00.542298 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:00.542294    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affvwpw"
Jul 28 20:52:00.542298 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:00.542308    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46dad53e6dbe1c59029ab6f83b7584462783dd085db80e8d7260c5a0557f5851"
Jul 28 20:52:07.103738 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.103702    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"]
Jul 28 20:52:07.104228 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.104020    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73f104ec-3e73-4db9-888c-0e1174583af1" containerName="extract"
Jul 28 20:52:07.104228 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.104031    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="73f104ec-3e73-4db9-888c-0e1174583af1" containerName="extract"
Jul 28 20:52:07.104228 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.104042    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73f104ec-3e73-4db9-888c-0e1174583af1" containerName="util"
Jul 28 20:52:07.104228 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.104047    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="73f104ec-3e73-4db9-888c-0e1174583af1" containerName="util"
Jul 28 20:52:07.104228 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.104065    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73f104ec-3e73-4db9-888c-0e1174583af1" containerName="pull"
Jul 28 20:52:07.104228 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.104071    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="73f104ec-3e73-4db9-888c-0e1174583af1" containerName="pull"
Jul 28 20:52:07.104228 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.104117    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="73f104ec-3e73-4db9-888c-0e1174583af1" containerName="extract"
Jul 28 20:52:07.108272 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.108253    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"
Jul 28 20:52:07.111301 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.111275    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 28 20:52:07.111617 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.111572    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 28 20:52:07.111617 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.111580    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 28 20:52:07.111801 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.111729    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sbqvl\""
Jul 28 20:52:07.112016 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.112003    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 28 20:52:07.129152 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.129124    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"]
Jul 28 20:52:07.222093 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.222055    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2545b75-760e-498e-b20c-f417090b67d0-tmp\") pod \"opendatahub-operator-controller-manager-6cc774576c-57mmd\" (UID: \"b2545b75-760e-498e-b20c-f417090b67d0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"
Jul 28 20:52:07.222278 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.222097    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b2545b75-760e-498e-b20c-f417090b67d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cc774576c-57mmd\" (UID: \"b2545b75-760e-498e-b20c-f417090b67d0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"
Jul 28 20:52:07.222278 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.222134    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lnfk\" (UniqueName: \"kubernetes.io/projected/b2545b75-760e-498e-b20c-f417090b67d0-kube-api-access-2lnfk\") pod \"opendatahub-operator-controller-manager-6cc774576c-57mmd\" (UID: \"b2545b75-760e-498e-b20c-f417090b67d0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"
Jul 28 20:52:07.222278 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.222155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2545b75-760e-498e-b20c-f417090b67d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cc774576c-57mmd\" (UID: \"b2545b75-760e-498e-b20c-f417090b67d0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"
Jul 28 20:52:07.222278 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.222173    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b2545b75-760e-498e-b20c-f417090b67d0-manifests\") pod \"opendatahub-operator-controller-manager-6cc774576c-57mmd\" (UID: \"b2545b75-760e-498e-b20c-f417090b67d0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"
Jul 28 20:52:07.322841 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.322805    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2545b75-760e-498e-b20c-f417090b67d0-tmp\") pod \"opendatahub-operator-controller-manager-6cc774576c-57mmd\" (UID: \"b2545b75-760e-498e-b20c-f417090b67d0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"
Jul 28 20:52:07.323011 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.322853    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b2545b75-760e-498e-b20c-f417090b67d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cc774576c-57mmd\" (UID: \"b2545b75-760e-498e-b20c-f417090b67d0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"
Jul 28 20:52:07.323060 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.323036    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lnfk\" (UniqueName: \"kubernetes.io/projected/b2545b75-760e-498e-b20c-f417090b67d0-kube-api-access-2lnfk\") pod \"opendatahub-operator-controller-manager-6cc774576c-57mmd\" (UID: \"b2545b75-760e-498e-b20c-f417090b67d0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"
Jul 28 20:52:07.323103 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.323078    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2545b75-760e-498e-b20c-f417090b67d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cc774576c-57mmd\" (UID: \"b2545b75-760e-498e-b20c-f417090b67d0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"
Jul 28 20:52:07.323160 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.323110    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b2545b75-760e-498e-b20c-f417090b67d0-manifests\") pod \"opendatahub-operator-controller-manager-6cc774576c-57mmd\" (UID: \"b2545b75-760e-498e-b20c-f417090b67d0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"
Jul 28 20:52:07.323520 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.323482    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b2545b75-760e-498e-b20c-f417090b67d0-manifests\") pod \"opendatahub-operator-controller-manager-6cc774576c-57mmd\" (UID: \"b2545b75-760e-498e-b20c-f417090b67d0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"
Jul 28 20:52:07.325173 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.325152    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2545b75-760e-498e-b20c-f417090b67d0-tmp\") pod \"opendatahub-operator-controller-manager-6cc774576c-57mmd\" (UID: \"b2545b75-760e-498e-b20c-f417090b67d0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"
Jul 28 20:52:07.325386 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.325364    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2545b75-760e-498e-b20c-f417090b67d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cc774576c-57mmd\" (UID: \"b2545b75-760e-498e-b20c-f417090b67d0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"
Jul 28 20:52:07.325459 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.325403    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b2545b75-760e-498e-b20c-f417090b67d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cc774576c-57mmd\" (UID: \"b2545b75-760e-498e-b20c-f417090b67d0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"
Jul 28 20:52:07.335076 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.335047    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lnfk\" (UniqueName: \"kubernetes.io/projected/b2545b75-760e-498e-b20c-f417090b67d0-kube-api-access-2lnfk\") pod \"opendatahub-operator-controller-manager-6cc774576c-57mmd\" (UID: \"b2545b75-760e-498e-b20c-f417090b67d0\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"
Jul 28 20:52:07.418185 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.418083    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"
Jul 28 20:52:07.589740 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:52:07.589706    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2545b75_760e_498e_b20c_f417090b67d0.slice/crio-4144851aa581d7c7b7c2a443ce9b2c6f893955e30058efcbd4f5be1338e39c60 WatchSource:0}: Error finding container 4144851aa581d7c7b7c2a443ce9b2c6f893955e30058efcbd4f5be1338e39c60: Status 404 returned error can't find the container with id 4144851aa581d7c7b7c2a443ce9b2c6f893955e30058efcbd4f5be1338e39c60
Jul 28 20:52:07.592310 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:07.592290    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"]
Jul 28 20:52:08.588339 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:08.588283    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd" event={"ID":"b2545b75-760e-498e-b20c-f417090b67d0","Type":"ContainerStarted","Data":"4144851aa581d7c7b7c2a443ce9b2c6f893955e30058efcbd4f5be1338e39c60"}
Jul 28 20:52:11.604926 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:11.604886    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd" event={"ID":"b2545b75-760e-498e-b20c-f417090b67d0","Type":"ContainerStarted","Data":"93c3aed1891cb316b9eb6b5daef4dd48e86980ddc289130f6a9bf62cb49ca2ea"}
Jul 28 20:52:12.610514 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:12.610478    2565 generic.go:356] "Generic (PLEG): container finished" podID="b2545b75-760e-498e-b20c-f417090b67d0" containerID="93c3aed1891cb316b9eb6b5daef4dd48e86980ddc289130f6a9bf62cb49ca2ea" exitCode=0
Jul 28 20:52:12.610907 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:12.610527    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd" event={"ID":"b2545b75-760e-498e-b20c-f417090b67d0","Type":"ContainerDied","Data":"93c3aed1891cb316b9eb6b5daef4dd48e86980ddc289130f6a9bf62cb49ca2ea"}
Jul 28 20:52:13.616659 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:13.616609    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd" event={"ID":"b2545b75-760e-498e-b20c-f417090b67d0","Type":"ContainerStarted","Data":"c904d5c91d954d0df05f9b6e794e120f4f41c0946b5bf2f4060836958a1cddb6"}
Jul 28 20:52:13.617036 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:13.616794    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"
Jul 28 20:52:13.634741 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:13.634689    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd" podStartSLOduration=2.10784234 podStartE2EDuration="6.634675723s" totalImagesPullingTime="3.505561443s" totalInitContainerRuntime="1.02127194s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:52:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:52:13.634113219 +0000 UTC m=+288.858456205" watchObservedRunningTime="2026-07-28 20:52:13.634675723 +0000 UTC m=+288.859018715"
Jul 28 20:52:24.623302 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:24.623274    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6cc774576c-57mmd"
Jul 28 20:52:25.207324 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:25.207295    2565 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 20:52:26.799257 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:26.799218    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68cb4f9b6-gxtld"]
Jul 28 20:52:26.802471 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:26.802455    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-gxtld"
Jul 28 20:52:26.806299 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:26.806272    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 28 20:52:26.806649 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:26.806612    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 28 20:52:26.806756 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:26.806691    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hc5bk\""
Jul 28 20:52:26.806870 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:26.806850    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 20:52:26.807020 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:26.807003    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 20:52:26.814759 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:26.814737    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68cb4f9b6-gxtld"]
Jul 28 20:52:26.868891 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:26.868864    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmjv6\" (UniqueName: \"kubernetes.io/projected/19df36da-d263-4e8f-8589-45e88f835635-kube-api-access-tmjv6\") pod \"kube-auth-proxy-68cb4f9b6-gxtld\" (UID: \"19df36da-d263-4e8f-8589-45e88f835635\") " pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-gxtld"
Jul 28 20:52:26.869064 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:26.868915    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19df36da-d263-4e8f-8589-45e88f835635-tmp\") pod \"kube-auth-proxy-68cb4f9b6-gxtld\" (UID: \"19df36da-d263-4e8f-8589-45e88f835635\") " pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-gxtld"
Jul 28 20:52:26.869064 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:26.868935    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19df36da-d263-4e8f-8589-45e88f835635-tls-certs\") pod \"kube-auth-proxy-68cb4f9b6-gxtld\" (UID: \"19df36da-d263-4e8f-8589-45e88f835635\") " pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-gxtld"
Jul 28 20:52:26.969855 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:26.969820    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmjv6\" (UniqueName: \"kubernetes.io/projected/19df36da-d263-4e8f-8589-45e88f835635-kube-api-access-tmjv6\") pod \"kube-auth-proxy-68cb4f9b6-gxtld\" (UID: \"19df36da-d263-4e8f-8589-45e88f835635\") " pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-gxtld"
Jul 28 20:52:26.970022 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:26.969879    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19df36da-d263-4e8f-8589-45e88f835635-tmp\") pod \"kube-auth-proxy-68cb4f9b6-gxtld\" (UID: \"19df36da-d263-4e8f-8589-45e88f835635\") " pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-gxtld"
Jul 28 20:52:26.970022 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:26.969903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19df36da-d263-4e8f-8589-45e88f835635-tls-certs\") pod \"kube-auth-proxy-68cb4f9b6-gxtld\" (UID: \"19df36da-d263-4e8f-8589-45e88f835635\") " pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-gxtld"
Jul 28 20:52:26.970022 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:52:26.969989    2565 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 28 20:52:26.970143 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:52:26.970048    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19df36da-d263-4e8f-8589-45e88f835635-tls-certs podName:19df36da-d263-4e8f-8589-45e88f835635 nodeName:}" failed. No retries permitted until 2026-07-28 20:52:27.470032054 +0000 UTC m=+302.694375018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/19df36da-d263-4e8f-8589-45e88f835635-tls-certs") pod "kube-auth-proxy-68cb4f9b6-gxtld" (UID: "19df36da-d263-4e8f-8589-45e88f835635") : secret "kube-auth-proxy-tls" not found
Jul 28 20:52:26.972076 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:26.972060    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19df36da-d263-4e8f-8589-45e88f835635-tmp\") pod \"kube-auth-proxy-68cb4f9b6-gxtld\" (UID: \"19df36da-d263-4e8f-8589-45e88f835635\") " pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-gxtld"
Jul 28 20:52:27.002980 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.002941    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmjv6\" (UniqueName: \"kubernetes.io/projected/19df36da-d263-4e8f-8589-45e88f835635-kube-api-access-tmjv6\") pod \"kube-auth-proxy-68cb4f9b6-gxtld\" (UID: \"19df36da-d263-4e8f-8589-45e88f835635\") " pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-gxtld"
Jul 28 20:52:27.084285 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.084211    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mdl2q"]
Jul 28 20:52:27.087508 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.087492    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mdl2q"
Jul 28 20:52:27.089981 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.089955    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 28 20:52:27.090154 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.090140    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-nmnph\""
Jul 28 20:52:27.107138 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.105781    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mdl2q"]
Jul 28 20:52:27.163083 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.163050    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg"]
Jul 28 20:52:27.166127 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.166108    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg"
Jul 28 20:52:27.168072 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.168051    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 20:52:27.168253 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.168240    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 20:52:27.168327 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.168265    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 20:52:27.168327 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.168265    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lh2n2\""
Jul 28 20:52:27.171858 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.171837    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv6wf\" (UniqueName: \"kubernetes.io/projected/c1831985-93ba-4383-bcb3-08079db37381-kube-api-access-vv6wf\") pod \"odh-model-controller-7998c67dc4-mdl2q\" (UID: \"c1831985-93ba-4383-bcb3-08079db37381\") " pod="opendatahub/odh-model-controller-7998c67dc4-mdl2q"
Jul 28 20:52:27.171944 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.171918    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1831985-93ba-4383-bcb3-08079db37381-cert\") pod \"odh-model-controller-7998c67dc4-mdl2q\" (UID: \"c1831985-93ba-4383-bcb3-08079db37381\") " pod="opendatahub/odh-model-controller-7998c67dc4-mdl2q"
Jul 28 20:52:27.174578 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.174555    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg"]
Jul 28 20:52:27.272920 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.272888    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv6wf\" (UniqueName: \"kubernetes.io/projected/c1831985-93ba-4383-bcb3-08079db37381-kube-api-access-vv6wf\") pod \"odh-model-controller-7998c67dc4-mdl2q\" (UID: \"c1831985-93ba-4383-bcb3-08079db37381\") " pod="opendatahub/odh-model-controller-7998c67dc4-mdl2q"
Jul 28 20:52:27.273090 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.272933    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lffhn\" (UniqueName: \"kubernetes.io/projected/eff5404c-a39a-4980-bf3b-88ea6665c1d8-kube-api-access-lffhn\") pod \"lws-controller-manager-bb9684f66-86jlg\" (UID: \"eff5404c-a39a-4980-bf3b-88ea6665c1d8\") " pod="openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg"
Jul 28 20:52:27.273090 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.272976    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eff5404c-a39a-4980-bf3b-88ea6665c1d8-cert\") pod \"lws-controller-manager-bb9684f66-86jlg\" (UID: \"eff5404c-a39a-4980-bf3b-88ea6665c1d8\") " pod="openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg"
Jul 28 20:52:27.273090 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.273002    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eff5404c-a39a-4980-bf3b-88ea6665c1d8-metrics-cert\") pod \"lws-controller-manager-bb9684f66-86jlg\" (UID: \"eff5404c-a39a-4980-bf3b-88ea6665c1d8\") " pod="openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg"
Jul 28 20:52:27.273090 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.273035    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1831985-93ba-4383-bcb3-08079db37381-cert\") pod \"odh-model-controller-7998c67dc4-mdl2q\" (UID: \"c1831985-93ba-4383-bcb3-08079db37381\") " pod="opendatahub/odh-model-controller-7998c67dc4-mdl2q"
Jul 28 20:52:27.273324 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:52:27.273111    2565 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 28 20:52:27.273324 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.273128    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eff5404c-a39a-4980-bf3b-88ea6665c1d8-manager-config\") pod \"lws-controller-manager-bb9684f66-86jlg\" (UID: \"eff5404c-a39a-4980-bf3b-88ea6665c1d8\") " pod="openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg"
Jul 28 20:52:27.273324 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:52:27.273165    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1831985-93ba-4383-bcb3-08079db37381-cert podName:c1831985-93ba-4383-bcb3-08079db37381 nodeName:}" failed. No retries permitted until 2026-07-28 20:52:27.773145473 +0000 UTC m=+302.997488439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c1831985-93ba-4383-bcb3-08079db37381-cert") pod "odh-model-controller-7998c67dc4-mdl2q" (UID: "c1831985-93ba-4383-bcb3-08079db37381") : secret "odh-model-controller-webhook-cert" not found
Jul 28 20:52:27.287244 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.287218    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv6wf\" (UniqueName: \"kubernetes.io/projected/c1831985-93ba-4383-bcb3-08079db37381-kube-api-access-vv6wf\") pod \"odh-model-controller-7998c67dc4-mdl2q\" (UID: \"c1831985-93ba-4383-bcb3-08079db37381\") " pod="opendatahub/odh-model-controller-7998c67dc4-mdl2q"
Jul 28 20:52:27.373676 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.373561    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eff5404c-a39a-4980-bf3b-88ea6665c1d8-manager-config\") pod \"lws-controller-manager-bb9684f66-86jlg\" (UID: \"eff5404c-a39a-4980-bf3b-88ea6665c1d8\") " pod="openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg"
Jul 28 20:52:27.373841 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.373679    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lffhn\" (UniqueName: \"kubernetes.io/projected/eff5404c-a39a-4980-bf3b-88ea6665c1d8-kube-api-access-lffhn\") pod \"lws-controller-manager-bb9684f66-86jlg\" (UID: \"eff5404c-a39a-4980-bf3b-88ea6665c1d8\") " pod="openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg"
Jul 28 20:52:27.373841 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.373720    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eff5404c-a39a-4980-bf3b-88ea6665c1d8-cert\") pod \"lws-controller-manager-bb9684f66-86jlg\" (UID: \"eff5404c-a39a-4980-bf3b-88ea6665c1d8\") " pod="openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg"
Jul 28 20:52:27.373841 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.373745    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eff5404c-a39a-4980-bf3b-88ea6665c1d8-metrics-cert\") pod \"lws-controller-manager-bb9684f66-86jlg\" (UID: \"eff5404c-a39a-4980-bf3b-88ea6665c1d8\") " pod="openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg"
Jul 28 20:52:27.374255 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.374231    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eff5404c-a39a-4980-bf3b-88ea6665c1d8-manager-config\") pod \"lws-controller-manager-bb9684f66-86jlg\" (UID: \"eff5404c-a39a-4980-bf3b-88ea6665c1d8\") " pod="openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg"
Jul 28 20:52:27.376080 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.376057    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eff5404c-a39a-4980-bf3b-88ea6665c1d8-cert\") pod \"lws-controller-manager-bb9684f66-86jlg\" (UID: \"eff5404c-a39a-4980-bf3b-88ea6665c1d8\") " pod="openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg"
Jul 28 20:52:27.376178 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.376160    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eff5404c-a39a-4980-bf3b-88ea6665c1d8-metrics-cert\") pod \"lws-controller-manager-bb9684f66-86jlg\" (UID: \"eff5404c-a39a-4980-bf3b-88ea6665c1d8\") " pod="openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg"
Jul 28 20:52:27.389767 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.389747    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lffhn\" (UniqueName: \"kubernetes.io/projected/eff5404c-a39a-4980-bf3b-88ea6665c1d8-kube-api-access-lffhn\") pod \"lws-controller-manager-bb9684f66-86jlg\" (UID: \"eff5404c-a39a-4980-bf3b-88ea6665c1d8\") " pod="openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg"
Jul 28 20:52:27.475023 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.474986    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19df36da-d263-4e8f-8589-45e88f835635-tls-certs\") pod \"kube-auth-proxy-68cb4f9b6-gxtld\" (UID: \"19df36da-d263-4e8f-8589-45e88f835635\") " pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-gxtld"
Jul 28 20:52:27.475213 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.475169    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg"
Jul 28 20:52:27.477306 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.477282    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19df36da-d263-4e8f-8589-45e88f835635-tls-certs\") pod \"kube-auth-proxy-68cb4f9b6-gxtld\" (UID: \"19df36da-d263-4e8f-8589-45e88f835635\") " pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-gxtld"
Jul 28 20:52:27.612446 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.612412    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg"]
Jul 28 20:52:27.615845 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:52:27.615821    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff5404c_a39a_4980_bf3b_88ea6665c1d8.slice/crio-8802eb446ae3f676a0320821a2d57073e214a9532db098f1e6a21bba9bf96933 WatchSource:0}: Error finding container 8802eb446ae3f676a0320821a2d57073e214a9532db098f1e6a21bba9bf96933: Status 404 returned error can't find the container with id 8802eb446ae3f676a0320821a2d57073e214a9532db098f1e6a21bba9bf96933
Jul 28 20:52:27.617823 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.617807    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:52:27.679988 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.679902    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg" event={"ID":"eff5404c-a39a-4980-bf3b-88ea6665c1d8","Type":"ContainerStarted","Data":"8802eb446ae3f676a0320821a2d57073e214a9532db098f1e6a21bba9bf96933"}
Jul 28 20:52:27.711985 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.711958    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-gxtld"
Jul 28 20:52:27.779146 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.779109    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1831985-93ba-4383-bcb3-08079db37381-cert\") pod \"odh-model-controller-7998c67dc4-mdl2q\" (UID: \"c1831985-93ba-4383-bcb3-08079db37381\") " pod="opendatahub/odh-model-controller-7998c67dc4-mdl2q"
Jul 28 20:52:27.781404 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.781373    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1831985-93ba-4383-bcb3-08079db37381-cert\") pod \"odh-model-controller-7998c67dc4-mdl2q\" (UID: \"c1831985-93ba-4383-bcb3-08079db37381\") " pod="opendatahub/odh-model-controller-7998c67dc4-mdl2q"
Jul 28 20:52:27.848891 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.848858    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68cb4f9b6-gxtld"]
Jul 28 20:52:27.850381 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:52:27.850353    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19df36da_d263_4e8f_8589_45e88f835635.slice/crio-41a89192be5dd84ede8e4e477cbeedbc2a54752ef9e99c76c8dd8a7531df4ee0 WatchSource:0}: Error finding container 41a89192be5dd84ede8e4e477cbeedbc2a54752ef9e99c76c8dd8a7531df4ee0: Status 404 returned error can't find the container with id 41a89192be5dd84ede8e4e477cbeedbc2a54752ef9e99c76c8dd8a7531df4ee0
Jul 28 20:52:27.997717 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:27.997689    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mdl2q"
Jul 28 20:52:28.153456 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:28.153428    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mdl2q"]
Jul 28 20:52:28.155526 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:52:28.155499    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1831985_93ba_4383_bcb3_08079db37381.slice/crio-ce8cf74e27cc949f9ffead2a6f1df8863474561999c284544a8dc2f6f31bb162 WatchSource:0}: Error finding container ce8cf74e27cc949f9ffead2a6f1df8863474561999c284544a8dc2f6f31bb162: Status 404 returned error can't find the container with id ce8cf74e27cc949f9ffead2a6f1df8863474561999c284544a8dc2f6f31bb162
Jul 28 20:52:28.687390 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:28.687347    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-gxtld" event={"ID":"19df36da-d263-4e8f-8589-45e88f835635","Type":"ContainerStarted","Data":"41a89192be5dd84ede8e4e477cbeedbc2a54752ef9e99c76c8dd8a7531df4ee0"}
Jul 28 20:52:28.688820 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:28.688786    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mdl2q" event={"ID":"c1831985-93ba-4383-bcb3-08079db37381","Type":"ContainerStarted","Data":"ce8cf74e27cc949f9ffead2a6f1df8863474561999c284544a8dc2f6f31bb162"}
Jul 28 20:52:29.698273 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:29.698234    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg" event={"ID":"eff5404c-a39a-4980-bf3b-88ea6665c1d8","Type":"ContainerStarted","Data":"6c8a9f98f191b6d58631edf04be7f6ca4e5f8a88753094e5a566183a927b8838"}
Jul 28 20:52:29.698786 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:29.698428    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg"
Jul 28 20:52:29.716484 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:29.716429    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg" podStartSLOduration=1.44468983 podStartE2EDuration="2.716414263s" totalImagesPullingTime="1.271724433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:52:29.713661061 +0000 UTC m=+304.938004046" watchObservedRunningTime="2026-07-28 20:52:29.716414263 +0000 UTC m=+304.940757228"
Jul 28 20:52:32.721158 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:32.721117    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-gxtld" event={"ID":"19df36da-d263-4e8f-8589-45e88f835635","Type":"ContainerStarted","Data":"d91e32da56d60cf41400f30733df6b2c1d032dd2b9c5f05b90b5a2b635c47299"}
Jul 28 20:52:32.722593 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:32.722562    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mdl2q" event={"ID":"c1831985-93ba-4383-bcb3-08079db37381","Type":"ContainerStarted","Data":"386ae6aed288ca674958c4d230eeff66024141ae9007efdb1646fc0a170a1100"}
Jul 28 20:52:32.722758 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:32.722688    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-mdl2q"
Jul 28 20:52:32.736937 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:32.736794    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68cb4f9b6-gxtld" podStartSLOduration=2.8431726299999998 podStartE2EDuration="6.736779856s" totalImagesPullingTime="3.893607226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:52:32.736043128 +0000 UTC m=+307.960386114" watchObservedRunningTime="2026-07-28 20:52:32.736779856 +0000 UTC m=+307.961122842"
Jul 28 20:52:32.752450 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:32.752390    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-mdl2q" podStartSLOduration=2.118789136 podStartE2EDuration="5.752367205s" totalImagesPullingTime="3.633578069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:52:32.751418197 +0000 UTC m=+307.975761185" watchObservedRunningTime="2026-07-28 20:52:32.752367205 +0000 UTC m=+307.976710193"
Jul 28 20:52:40.707201 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:40.707167    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-bb9684f66-86jlg"
Jul 28 20:52:43.729838 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:52:43.729809    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-mdl2q"
Jul 28 20:53:43.267094 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:43.267009    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz"]
Jul 28 20:53:43.270724 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:43.270707    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz"
Jul 28 20:53:43.272895 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:43.272874    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 20:53:43.272996 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:43.272875    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 20:53:43.273129 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:43.273115    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-m9l2k\""
Jul 28 20:53:43.278153 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:43.278135    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz"]
Jul 28 20:53:43.383373 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:43.383341    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz\" (UID: \"7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz"
Jul 28 20:53:43.383373 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:43.383382    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz\" (UID: \"7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz"
Jul 28 20:53:43.383708 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:43.383576    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh2r4\" (UniqueName: \"kubernetes.io/projected/7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4-kube-api-access-jh2r4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz\" (UID: \"7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz"
Jul 28 20:53:43.484762 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:43.484722    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2r4\" (UniqueName: \"kubernetes.io/projected/7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4-kube-api-access-jh2r4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz\" (UID: \"7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz"
Jul 28 20:53:43.484944 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:43.484781    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz\" (UID: \"7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz"
Jul 28 20:53:43.484944 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:43.484808    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz\" (UID: \"7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz"
Jul 28 20:53:43.485180 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:43.485158    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz\" (UID: \"7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz"
Jul 28 20:53:43.485236 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:43.485185    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz\" (UID: \"7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz"
Jul 28 20:53:43.493427 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:43.493398    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh2r4\" (UniqueName: \"kubernetes.io/projected/7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4-kube-api-access-jh2r4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz\" (UID: \"7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz"
Jul 28 20:53:43.580668 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:43.580546    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz"
Jul 28 20:53:43.709614 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:43.709582    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz"]
Jul 28 20:53:43.712656 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:53:43.712617    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ffaf5af_3b4e_41e8_9e3e_bb3490d559b4.slice/crio-6946dedcd5b3d312d66a6baa972c22fa2fd3199877977f099e132bf4c942f513 WatchSource:0}: Error finding container 6946dedcd5b3d312d66a6baa972c22fa2fd3199877977f099e132bf4c942f513: Status 404 returned error can't find the container with id 6946dedcd5b3d312d66a6baa972c22fa2fd3199877977f099e132bf4c942f513
Jul 28 20:53:44.053774 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.053741    2565 generic.go:356] "Generic (PLEG): container finished" podID="7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4" containerID="3ee8abc66e3ce9924111f31ba399486365acccf79a9eceb46c6ad80751a2fd8d" exitCode=0
Jul 28 20:53:44.053976 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.053826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz" event={"ID":"7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4","Type":"ContainerDied","Data":"3ee8abc66e3ce9924111f31ba399486365acccf79a9eceb46c6ad80751a2fd8d"}
Jul 28 20:53:44.053976 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.053872    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz" event={"ID":"7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4","Type":"ContainerStarted","Data":"6946dedcd5b3d312d66a6baa972c22fa2fd3199877977f099e132bf4c942f513"}
Jul 28 20:53:44.057663 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.057624    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn"]
Jul 28 20:53:44.060966 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.060951    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn"
Jul 28 20:53:44.071416 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.071393    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn"]
Jul 28 20:53:44.089451 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.089426    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/073b3df5-255a-43d8-bdf8-1e75b767507a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn\" (UID: \"073b3df5-255a-43d8-bdf8-1e75b767507a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn"
Jul 28 20:53:44.089556 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.089463    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhjhs\" (UniqueName: \"kubernetes.io/projected/073b3df5-255a-43d8-bdf8-1e75b767507a-kube-api-access-lhjhs\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn\" (UID: \"073b3df5-255a-43d8-bdf8-1e75b767507a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn"
Jul 28 20:53:44.089556 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.089487    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/073b3df5-255a-43d8-bdf8-1e75b767507a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn\" (UID: \"073b3df5-255a-43d8-bdf8-1e75b767507a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn"
Jul 28 20:53:44.190746 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.190706    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/073b3df5-255a-43d8-bdf8-1e75b767507a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn\" (UID: \"073b3df5-255a-43d8-bdf8-1e75b767507a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn"
Jul 28 20:53:44.190945 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.190779    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/073b3df5-255a-43d8-bdf8-1e75b767507a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn\" (UID: \"073b3df5-255a-43d8-bdf8-1e75b767507a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn"
Jul 28 20:53:44.190945 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.190805    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhjhs\" (UniqueName: \"kubernetes.io/projected/073b3df5-255a-43d8-bdf8-1e75b767507a-kube-api-access-lhjhs\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn\" (UID: \"073b3df5-255a-43d8-bdf8-1e75b767507a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn"
Jul 28 20:53:44.191106 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.191082    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/073b3df5-255a-43d8-bdf8-1e75b767507a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn\" (UID: \"073b3df5-255a-43d8-bdf8-1e75b767507a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn"
Jul 28 20:53:44.191171 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.191143    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/073b3df5-255a-43d8-bdf8-1e75b767507a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn\" (UID: \"073b3df5-255a-43d8-bdf8-1e75b767507a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn"
Jul 28 20:53:44.199983 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.199958    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhjhs\" (UniqueName: \"kubernetes.io/projected/073b3df5-255a-43d8-bdf8-1e75b767507a-kube-api-access-lhjhs\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn\" (UID: \"073b3df5-255a-43d8-bdf8-1e75b767507a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn"
Jul 28 20:53:44.369853 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.369772    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn"
Jul 28 20:53:44.502527 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.502497    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn"]
Jul 28 20:53:44.504584 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:53:44.504556    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073b3df5_255a_43d8_bdf8_1e75b767507a.slice/crio-f7484cadbb5d8fbf9772fe53b6df0b2acd30f0a2a2cb2cb3a37ac07601e12960 WatchSource:0}: Error finding container f7484cadbb5d8fbf9772fe53b6df0b2acd30f0a2a2cb2cb3a37ac07601e12960: Status 404 returned error can't find the container with id f7484cadbb5d8fbf9772fe53b6df0b2acd30f0a2a2cb2cb3a37ac07601e12960
Jul 28 20:53:44.661830 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.661748    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn"]
Jul 28 20:53:44.665209 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.665184    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn"
Jul 28 20:53:44.675058 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.675030    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn"]
Jul 28 20:53:44.694810 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.694784    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2hss\" (UniqueName: \"kubernetes.io/projected/ed38cb3e-13af-4f80-b366-065ffe40d3f4-kube-api-access-b2hss\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn\" (UID: \"ed38cb3e-13af-4f80-b366-065ffe40d3f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn"
Jul 28 20:53:44.694937 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.694843    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed38cb3e-13af-4f80-b366-065ffe40d3f4-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn\" (UID: \"ed38cb3e-13af-4f80-b366-065ffe40d3f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn"
Jul 28 20:53:44.694937 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.694919    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed38cb3e-13af-4f80-b366-065ffe40d3f4-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn\" (UID: \"ed38cb3e-13af-4f80-b366-065ffe40d3f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn"
Jul 28 20:53:44.796333 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.796291    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2hss\" (UniqueName: \"kubernetes.io/projected/ed38cb3e-13af-4f80-b366-065ffe40d3f4-kube-api-access-b2hss\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn\" (UID: \"ed38cb3e-13af-4f80-b366-065ffe40d3f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn"
Jul 28 20:53:44.796544 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.796361    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed38cb3e-13af-4f80-b366-065ffe40d3f4-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn\" (UID: \"ed38cb3e-13af-4f80-b366-065ffe40d3f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn"
Jul 28 20:53:44.796544 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.796405    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed38cb3e-13af-4f80-b366-065ffe40d3f4-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn\" (UID: \"ed38cb3e-13af-4f80-b366-065ffe40d3f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn"
Jul 28 20:53:44.796847 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.796823    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed38cb3e-13af-4f80-b366-065ffe40d3f4-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn\" (UID: \"ed38cb3e-13af-4f80-b366-065ffe40d3f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn"
Jul 28 20:53:44.796935 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.796900    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed38cb3e-13af-4f80-b366-065ffe40d3f4-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn\" (UID: \"ed38cb3e-13af-4f80-b366-065ffe40d3f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn"
Jul 28 20:53:44.804811 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.804780    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2hss\" (UniqueName: \"kubernetes.io/projected/ed38cb3e-13af-4f80-b366-065ffe40d3f4-kube-api-access-b2hss\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn\" (UID: \"ed38cb3e-13af-4f80-b366-065ffe40d3f4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn"
Jul 28 20:53:44.974118 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:44.974082    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn"
Jul 28 20:53:45.061096 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:45.061064    2565 generic.go:356] "Generic (PLEG): container finished" podID="073b3df5-255a-43d8-bdf8-1e75b767507a" containerID="4eea0193545d771642faca8c4e603a2fa5d3cf719624955da8f6ae070b485d9f" exitCode=0
Jul 28 20:53:45.061240 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:45.061133    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn" event={"ID":"073b3df5-255a-43d8-bdf8-1e75b767507a","Type":"ContainerDied","Data":"4eea0193545d771642faca8c4e603a2fa5d3cf719624955da8f6ae070b485d9f"}
Jul 28 20:53:45.061240 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:45.061183    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn" event={"ID":"073b3df5-255a-43d8-bdf8-1e75b767507a","Type":"ContainerStarted","Data":"f7484cadbb5d8fbf9772fe53b6df0b2acd30f0a2a2cb2cb3a37ac07601e12960"}
Jul 28 20:53:45.131521 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:45.131496    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn"]
Jul 28 20:53:45.133818 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:53:45.133789    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded38cb3e_13af_4f80_b366_065ffe40d3f4.slice/crio-5493df557e41873675a52c6d63eddc1d2d8e7726254f3088ced0f0cf5f3cca8e WatchSource:0}: Error finding container 5493df557e41873675a52c6d63eddc1d2d8e7726254f3088ced0f0cf5f3cca8e: Status 404 returned error can't find the container with id 5493df557e41873675a52c6d63eddc1d2d8e7726254f3088ced0f0cf5f3cca8e
Jul 28 20:53:45.266311 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:45.266276    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j"]
Jul 28 20:53:45.269664 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:45.269621    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j"
Jul 28 20:53:45.279744 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:45.279717    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j"]
Jul 28 20:53:45.301158 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:45.301131    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51aa2b30-ff05-45fa-90db-5aa6bf4970ae-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j\" (UID: \"51aa2b30-ff05-45fa-90db-5aa6bf4970ae\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j"
Jul 28 20:53:45.301294 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:45.301189    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51aa2b30-ff05-45fa-90db-5aa6bf4970ae-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j\" (UID: \"51aa2b30-ff05-45fa-90db-5aa6bf4970ae\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j"
Jul 28 20:53:45.301294 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:45.301262    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfmtm\" (UniqueName: \"kubernetes.io/projected/51aa2b30-ff05-45fa-90db-5aa6bf4970ae-kube-api-access-vfmtm\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j\" (UID: \"51aa2b30-ff05-45fa-90db-5aa6bf4970ae\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j"
Jul 28 20:53:45.401991 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:45.401951    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51aa2b30-ff05-45fa-90db-5aa6bf4970ae-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j\" (UID: \"51aa2b30-ff05-45fa-90db-5aa6bf4970ae\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j"
Jul 28 20:53:45.402386 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:45.402016    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51aa2b30-ff05-45fa-90db-5aa6bf4970ae-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j\" (UID: \"51aa2b30-ff05-45fa-90db-5aa6bf4970ae\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j"
Jul 28 20:53:45.402386 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:45.402047    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfmtm\" (UniqueName: \"kubernetes.io/projected/51aa2b30-ff05-45fa-90db-5aa6bf4970ae-kube-api-access-vfmtm\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j\" (UID: \"51aa2b30-ff05-45fa-90db-5aa6bf4970ae\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j"
Jul 28 20:53:45.402386 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:45.402343    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51aa2b30-ff05-45fa-90db-5aa6bf4970ae-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j\" (UID: \"51aa2b30-ff05-45fa-90db-5aa6bf4970ae\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j"
Jul 28 20:53:45.402386 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:45.402361    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51aa2b30-ff05-45fa-90db-5aa6bf4970ae-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j\" (UID: \"51aa2b30-ff05-45fa-90db-5aa6bf4970ae\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j"
Jul 28 20:53:45.409846 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:45.409815    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfmtm\" (UniqueName: \"kubernetes.io/projected/51aa2b30-ff05-45fa-90db-5aa6bf4970ae-kube-api-access-vfmtm\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j\" (UID: \"51aa2b30-ff05-45fa-90db-5aa6bf4970ae\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j"
Jul 28 20:53:45.600611 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:45.600539    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j"
Jul 28 20:53:45.810787 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:45.810757    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j"]
Jul 28 20:53:45.898495 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:53:45.898408    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51aa2b30_ff05_45fa_90db_5aa6bf4970ae.slice/crio-a0ec841c2ce1198eb4627f4fbcb54a0e893ba078a5befd3ca320c24cbbb51204 WatchSource:0}: Error finding container a0ec841c2ce1198eb4627f4fbcb54a0e893ba078a5befd3ca320c24cbbb51204: Status 404 returned error can't find the container with id a0ec841c2ce1198eb4627f4fbcb54a0e893ba078a5befd3ca320c24cbbb51204
Jul 28 20:53:46.068262 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:46.068229    2565 generic.go:356] "Generic (PLEG): container finished" podID="7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4" containerID="14aaca83c6830010c3d2c989f89d8fecb37577dfb9435bb17e366178a8d44a88" exitCode=0
Jul 28 20:53:46.068445 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:46.068314    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz" event={"ID":"7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4","Type":"ContainerDied","Data":"14aaca83c6830010c3d2c989f89d8fecb37577dfb9435bb17e366178a8d44a88"}
Jul 28 20:53:46.070244 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:46.070220    2565 generic.go:356] "Generic (PLEG): container finished" podID="073b3df5-255a-43d8-bdf8-1e75b767507a" containerID="5cafec4d7f0c419d456b9a45b4da7fc28376e425d892ae24e0438c5dfb1a3abc" exitCode=0
Jul 28 20:53:46.070358 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:46.070303    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn" event={"ID":"073b3df5-255a-43d8-bdf8-1e75b767507a","Type":"ContainerDied","Data":"5cafec4d7f0c419d456b9a45b4da7fc28376e425d892ae24e0438c5dfb1a3abc"}
Jul 28 20:53:46.071714 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:46.071691    2565 generic.go:356] "Generic (PLEG): container finished" podID="51aa2b30-ff05-45fa-90db-5aa6bf4970ae" containerID="885d3d5f71fa316f7878ff24ca3dfa67922efb4af5c42f6896eea6c17a5f807c" exitCode=0
Jul 28 20:53:46.071811 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:46.071760    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j" event={"ID":"51aa2b30-ff05-45fa-90db-5aa6bf4970ae","Type":"ContainerDied","Data":"885d3d5f71fa316f7878ff24ca3dfa67922efb4af5c42f6896eea6c17a5f807c"}
Jul 28 20:53:46.071811 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:46.071792    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j" event={"ID":"51aa2b30-ff05-45fa-90db-5aa6bf4970ae","Type":"ContainerStarted","Data":"a0ec841c2ce1198eb4627f4fbcb54a0e893ba078a5befd3ca320c24cbbb51204"}
Jul 28 20:53:46.073453 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:46.073350    2565 generic.go:356] "Generic (PLEG): container finished" podID="ed38cb3e-13af-4f80-b366-065ffe40d3f4" containerID="bf5798ae16fa8f4aa6b0b50b76188dcd22babe107466803d575bc2f70b593139" exitCode=0
Jul 28 20:53:46.073453 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:46.073386    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn" event={"ID":"ed38cb3e-13af-4f80-b366-065ffe40d3f4","Type":"ContainerDied","Data":"bf5798ae16fa8f4aa6b0b50b76188dcd22babe107466803d575bc2f70b593139"}
Jul 28 20:53:46.073453 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:46.073418    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn" event={"ID":"ed38cb3e-13af-4f80-b366-065ffe40d3f4","Type":"ContainerStarted","Data":"5493df557e41873675a52c6d63eddc1d2d8e7726254f3088ced0f0cf5f3cca8e"}
Jul 28 20:53:46.270821 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:53:46.270789    2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073b3df5_255a_43d8_bdf8_1e75b767507a.slice/crio-conmon-0232f0fe83343e82294a9684f56e8fdb15fb14c91ab617511018d545d23456e5.scope\": RecentStats: unable to find data in memory cache]"
Jul 28 20:53:47.080071 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:47.080035    2565 generic.go:356] "Generic (PLEG): container finished" podID="7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4" containerID="261c7c907ba12908394cc14ebb20c82813144fb0ecca1c61b7a211efbfb95527" exitCode=0
Jul 28 20:53:47.080501 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:47.080113    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz" event={"ID":"7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4","Type":"ContainerDied","Data":"261c7c907ba12908394cc14ebb20c82813144fb0ecca1c61b7a211efbfb95527"}
Jul 28 20:53:47.082176 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:47.082157    2565 generic.go:356] "Generic (PLEG): container finished" podID="073b3df5-255a-43d8-bdf8-1e75b767507a" containerID="0232f0fe83343e82294a9684f56e8fdb15fb14c91ab617511018d545d23456e5" exitCode=0
Jul 28 20:53:47.082275 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:47.082239    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn" event={"ID":"073b3df5-255a-43d8-bdf8-1e75b767507a","Type":"ContainerDied","Data":"0232f0fe83343e82294a9684f56e8fdb15fb14c91ab617511018d545d23456e5"}
Jul 28 20:53:47.084125 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:47.084105    2565 generic.go:356] "Generic (PLEG): container finished" podID="51aa2b30-ff05-45fa-90db-5aa6bf4970ae" containerID="7b72694872f600768f3c285869028521efda11170f54a384eea9279d676ee20c" exitCode=0
Jul 28 20:53:47.084227 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:47.084170    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j" event={"ID":"51aa2b30-ff05-45fa-90db-5aa6bf4970ae","Type":"ContainerDied","Data":"7b72694872f600768f3c285869028521efda11170f54a384eea9279d676ee20c"}
Jul 28 20:53:47.085946 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:47.085925    2565 generic.go:356] "Generic (PLEG): container finished" podID="ed38cb3e-13af-4f80-b366-065ffe40d3f4" containerID="dc4c91f54c393582e3bf957b249494bd0673e46559087dc9c747671dbcf5fd83" exitCode=0
Jul 28 20:53:47.086053 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:47.086003    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn" event={"ID":"ed38cb3e-13af-4f80-b366-065ffe40d3f4","Type":"ContainerDied","Data":"dc4c91f54c393582e3bf957b249494bd0673e46559087dc9c747671dbcf5fd83"}
Jul 28 20:53:48.093101 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.093066    2565 generic.go:356] "Generic (PLEG): container finished" podID="51aa2b30-ff05-45fa-90db-5aa6bf4970ae" containerID="7641762cad26e0249f92ce966b9e92347ae58c7e1f29466d5c0954557620bee1" exitCode=0
Jul 28 20:53:48.093514 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.093147    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j" event={"ID":"51aa2b30-ff05-45fa-90db-5aa6bf4970ae","Type":"ContainerDied","Data":"7641762cad26e0249f92ce966b9e92347ae58c7e1f29466d5c0954557620bee1"}
Jul 28 20:53:48.095130 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.095105    2565 generic.go:356] "Generic (PLEG): container finished" podID="ed38cb3e-13af-4f80-b366-065ffe40d3f4" containerID="32e180cf67b103f59fb120fc714d0fa57198445875b395b0660ce8a76200b939" exitCode=0
Jul 28 20:53:48.095255 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.095183    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn" event={"ID":"ed38cb3e-13af-4f80-b366-065ffe40d3f4","Type":"ContainerDied","Data":"32e180cf67b103f59fb120fc714d0fa57198445875b395b0660ce8a76200b939"}
Jul 28 20:53:48.226057 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.226030    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz"
Jul 28 20:53:48.254895 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.254874    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn"
Jul 28 20:53:48.331380 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.331341    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhjhs\" (UniqueName: \"kubernetes.io/projected/073b3df5-255a-43d8-bdf8-1e75b767507a-kube-api-access-lhjhs\") pod \"073b3df5-255a-43d8-bdf8-1e75b767507a\" (UID: \"073b3df5-255a-43d8-bdf8-1e75b767507a\") "
Jul 28 20:53:48.331380 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.331386    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4-bundle\") pod \"7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4\" (UID: \"7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4\") "
Jul 28 20:53:48.331659 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.331439    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/073b3df5-255a-43d8-bdf8-1e75b767507a-bundle\") pod \"073b3df5-255a-43d8-bdf8-1e75b767507a\" (UID: \"073b3df5-255a-43d8-bdf8-1e75b767507a\") "
Jul 28 20:53:48.331659 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.331456    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4-util\") pod \"7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4\" (UID: \"7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4\") "
Jul 28 20:53:48.331659 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.331477    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/073b3df5-255a-43d8-bdf8-1e75b767507a-util\") pod \"073b3df5-255a-43d8-bdf8-1e75b767507a\" (UID: \"073b3df5-255a-43d8-bdf8-1e75b767507a\") "
Jul 28 20:53:48.331659 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.331521    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh2r4\" (UniqueName: \"kubernetes.io/projected/7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4-kube-api-access-jh2r4\") pod \"7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4\" (UID: \"7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4\") "
Jul 28 20:53:48.332082 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.332051    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4-bundle" (OuterVolumeSpecName: "bundle") pod "7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4" (UID: "7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:53:48.332205 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.332163    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073b3df5-255a-43d8-bdf8-1e75b767507a-bundle" (OuterVolumeSpecName: "bundle") pod "073b3df5-255a-43d8-bdf8-1e75b767507a" (UID: "073b3df5-255a-43d8-bdf8-1e75b767507a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:53:48.333996 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.333972    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4-kube-api-access-jh2r4" (OuterVolumeSpecName: "kube-api-access-jh2r4") pod "7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4" (UID: "7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4"). InnerVolumeSpecName "kube-api-access-jh2r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:53:48.334091 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.334000    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073b3df5-255a-43d8-bdf8-1e75b767507a-kube-api-access-lhjhs" (OuterVolumeSpecName: "kube-api-access-lhjhs") pod "073b3df5-255a-43d8-bdf8-1e75b767507a" (UID: "073b3df5-255a-43d8-bdf8-1e75b767507a"). InnerVolumeSpecName "kube-api-access-lhjhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:53:48.337449 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.337413    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073b3df5-255a-43d8-bdf8-1e75b767507a-util" (OuterVolumeSpecName: "util") pod "073b3df5-255a-43d8-bdf8-1e75b767507a" (UID: "073b3df5-255a-43d8-bdf8-1e75b767507a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:53:48.337757 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.337738    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4-util" (OuterVolumeSpecName: "util") pod "7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4" (UID: "7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:53:48.433063 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.432991    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhjhs\" (UniqueName: \"kubernetes.io/projected/073b3df5-255a-43d8-bdf8-1e75b767507a-kube-api-access-lhjhs\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:53:48.433063 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.433018    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4-bundle\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:53:48.433063 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.433028    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/073b3df5-255a-43d8-bdf8-1e75b767507a-bundle\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:53:48.433063 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.433036    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4-util\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:53:48.433063 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.433044    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/073b3df5-255a-43d8-bdf8-1e75b767507a-util\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:53:48.433063 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:48.433052    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh2r4\" (UniqueName: \"kubernetes.io/projected/7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4-kube-api-access-jh2r4\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:53:49.102485 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.102454    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz"
Jul 28 20:53:49.102485 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.102470    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz" event={"ID":"7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4","Type":"ContainerDied","Data":"6946dedcd5b3d312d66a6baa972c22fa2fd3199877977f099e132bf4c942f513"}
Jul 28 20:53:49.102986 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.102514    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6946dedcd5b3d312d66a6baa972c22fa2fd3199877977f099e132bf4c942f513"
Jul 28 20:53:49.104294 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.104274    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn"
Jul 28 20:53:49.104416 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.104294    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn" event={"ID":"073b3df5-255a-43d8-bdf8-1e75b767507a","Type":"ContainerDied","Data":"f7484cadbb5d8fbf9772fe53b6df0b2acd30f0a2a2cb2cb3a37ac07601e12960"}
Jul 28 20:53:49.104416 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.104335    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7484cadbb5d8fbf9772fe53b6df0b2acd30f0a2a2cb2cb3a37ac07601e12960"
Jul 28 20:53:49.245136 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.245108    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn"
Jul 28 20:53:49.270130 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.270105    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j"
Jul 28 20:53:49.340497 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.340454    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfmtm\" (UniqueName: \"kubernetes.io/projected/51aa2b30-ff05-45fa-90db-5aa6bf4970ae-kube-api-access-vfmtm\") pod \"51aa2b30-ff05-45fa-90db-5aa6bf4970ae\" (UID: \"51aa2b30-ff05-45fa-90db-5aa6bf4970ae\") "
Jul 28 20:53:49.340672 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.340543    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51aa2b30-ff05-45fa-90db-5aa6bf4970ae-bundle\") pod \"51aa2b30-ff05-45fa-90db-5aa6bf4970ae\" (UID: \"51aa2b30-ff05-45fa-90db-5aa6bf4970ae\") "
Jul 28 20:53:49.340672 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.340569    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed38cb3e-13af-4f80-b366-065ffe40d3f4-util\") pod \"ed38cb3e-13af-4f80-b366-065ffe40d3f4\" (UID: \"ed38cb3e-13af-4f80-b366-065ffe40d3f4\") "
Jul 28 20:53:49.340672 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.340596    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2hss\" (UniqueName: \"kubernetes.io/projected/ed38cb3e-13af-4f80-b366-065ffe40d3f4-kube-api-access-b2hss\") pod \"ed38cb3e-13af-4f80-b366-065ffe40d3f4\" (UID: \"ed38cb3e-13af-4f80-b366-065ffe40d3f4\") "
Jul 28 20:53:49.340672 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.340617    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed38cb3e-13af-4f80-b366-065ffe40d3f4-bundle\") pod \"ed38cb3e-13af-4f80-b366-065ffe40d3f4\" (UID: \"ed38cb3e-13af-4f80-b366-065ffe40d3f4\") "
Jul 28 20:53:49.340672 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.340653    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51aa2b30-ff05-45fa-90db-5aa6bf4970ae-util\") pod \"51aa2b30-ff05-45fa-90db-5aa6bf4970ae\" (UID: \"51aa2b30-ff05-45fa-90db-5aa6bf4970ae\") "
Jul 28 20:53:49.341125 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.341092    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51aa2b30-ff05-45fa-90db-5aa6bf4970ae-bundle" (OuterVolumeSpecName: "bundle") pod "51aa2b30-ff05-45fa-90db-5aa6bf4970ae" (UID: "51aa2b30-ff05-45fa-90db-5aa6bf4970ae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:53:49.341273 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.341253    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed38cb3e-13af-4f80-b366-065ffe40d3f4-bundle" (OuterVolumeSpecName: "bundle") pod "ed38cb3e-13af-4f80-b366-065ffe40d3f4" (UID: "ed38cb3e-13af-4f80-b366-065ffe40d3f4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:53:49.342888 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.342858    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed38cb3e-13af-4f80-b366-065ffe40d3f4-kube-api-access-b2hss" (OuterVolumeSpecName: "kube-api-access-b2hss") pod "ed38cb3e-13af-4f80-b366-065ffe40d3f4" (UID: "ed38cb3e-13af-4f80-b366-065ffe40d3f4"). InnerVolumeSpecName "kube-api-access-b2hss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:53:49.342958 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.342933    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51aa2b30-ff05-45fa-90db-5aa6bf4970ae-kube-api-access-vfmtm" (OuterVolumeSpecName: "kube-api-access-vfmtm") pod "51aa2b30-ff05-45fa-90db-5aa6bf4970ae" (UID: "51aa2b30-ff05-45fa-90db-5aa6bf4970ae"). InnerVolumeSpecName "kube-api-access-vfmtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:53:49.346108 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.346074    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed38cb3e-13af-4f80-b366-065ffe40d3f4-util" (OuterVolumeSpecName: "util") pod "ed38cb3e-13af-4f80-b366-065ffe40d3f4" (UID: "ed38cb3e-13af-4f80-b366-065ffe40d3f4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:53:49.347163 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.347144    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51aa2b30-ff05-45fa-90db-5aa6bf4970ae-util" (OuterVolumeSpecName: "util") pod "51aa2b30-ff05-45fa-90db-5aa6bf4970ae" (UID: "51aa2b30-ff05-45fa-90db-5aa6bf4970ae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:53:49.441511 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.441481    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51aa2b30-ff05-45fa-90db-5aa6bf4970ae-bundle\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:53:49.441511 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.441511    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed38cb3e-13af-4f80-b366-065ffe40d3f4-util\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:53:49.441747 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.441524    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2hss\" (UniqueName: \"kubernetes.io/projected/ed38cb3e-13af-4f80-b366-065ffe40d3f4-kube-api-access-b2hss\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:53:49.441747 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.441537    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed38cb3e-13af-4f80-b366-065ffe40d3f4-bundle\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:53:49.441747 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.441548    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51aa2b30-ff05-45fa-90db-5aa6bf4970ae-util\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:53:49.441747 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:49.441559    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfmtm\" (UniqueName: \"kubernetes.io/projected/51aa2b30-ff05-45fa-90db-5aa6bf4970ae-kube-api-access-vfmtm\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:53:50.111280 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:50.111238    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j" event={"ID":"51aa2b30-ff05-45fa-90db-5aa6bf4970ae","Type":"ContainerDied","Data":"a0ec841c2ce1198eb4627f4fbcb54a0e893ba078a5befd3ca320c24cbbb51204"}
Jul 28 20:53:50.111731 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:50.111251    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j"
Jul 28 20:53:50.111731 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:50.111332    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0ec841c2ce1198eb4627f4fbcb54a0e893ba078a5befd3ca320c24cbbb51204"
Jul 28 20:53:50.113198 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:50.113172    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn" event={"ID":"ed38cb3e-13af-4f80-b366-065ffe40d3f4","Type":"ContainerDied","Data":"5493df557e41873675a52c6d63eddc1d2d8e7726254f3088ced0f0cf5f3cca8e"}
Jul 28 20:53:50.113343 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:50.113202    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn"
Jul 28 20:53:50.113343 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:53:50.113206    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5493df557e41873675a52c6d63eddc1d2d8e7726254f3088ced0f0cf5f3cca8e"
Jul 28 20:54:07.014449 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014406    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-6zq5m"]
Jul 28 20:54:07.014821 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014769    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="073b3df5-255a-43d8-bdf8-1e75b767507a" containerName="extract"
Jul 28 20:54:07.014821 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014782    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="073b3df5-255a-43d8-bdf8-1e75b767507a" containerName="extract"
Jul 28 20:54:07.014821 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014795    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4" containerName="util"
Jul 28 20:54:07.014821 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014801    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4" containerName="util"
Jul 28 20:54:07.014821 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014809    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="073b3df5-255a-43d8-bdf8-1e75b767507a" containerName="util"
Jul 28 20:54:07.014821 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014814    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="073b3df5-255a-43d8-bdf8-1e75b767507a" containerName="util"
Jul 28 20:54:07.014821 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014819    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed38cb3e-13af-4f80-b366-065ffe40d3f4" containerName="pull"
Jul 28 20:54:07.014821 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014823    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed38cb3e-13af-4f80-b366-065ffe40d3f4" containerName="pull"
Jul 28 20:54:07.015069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014831    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed38cb3e-13af-4f80-b366-065ffe40d3f4" containerName="util"
Jul 28 20:54:07.015069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014836    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed38cb3e-13af-4f80-b366-065ffe40d3f4" containerName="util"
Jul 28 20:54:07.015069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014843    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4" containerName="pull"
Jul 28 20:54:07.015069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014848    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4" containerName="pull"
Jul 28 20:54:07.015069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014855    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51aa2b30-ff05-45fa-90db-5aa6bf4970ae" containerName="pull"
Jul 28 20:54:07.015069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014859    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="51aa2b30-ff05-45fa-90db-5aa6bf4970ae" containerName="pull"
Jul 28 20:54:07.015069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014868    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="073b3df5-255a-43d8-bdf8-1e75b767507a" containerName="pull"
Jul 28 20:54:07.015069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014873    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="073b3df5-255a-43d8-bdf8-1e75b767507a" containerName="pull"
Jul 28 20:54:07.015069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014878    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed38cb3e-13af-4f80-b366-065ffe40d3f4" containerName="extract"
Jul 28 20:54:07.015069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014883    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed38cb3e-13af-4f80-b366-065ffe40d3f4" containerName="extract"
Jul 28 20:54:07.015069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014889    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51aa2b30-ff05-45fa-90db-5aa6bf4970ae" containerName="util"
Jul 28 20:54:07.015069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014893    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="51aa2b30-ff05-45fa-90db-5aa6bf4970ae" containerName="util"
Jul 28 20:54:07.015069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014898    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51aa2b30-ff05-45fa-90db-5aa6bf4970ae" containerName="extract"
Jul 28 20:54:07.015069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014902    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="51aa2b30-ff05-45fa-90db-5aa6bf4970ae" containerName="extract"
Jul 28 20:54:07.015069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014911    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4" containerName="extract"
Jul 28 20:54:07.015069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014916    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4" containerName="extract"
Jul 28 20:54:07.015069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014966    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="51aa2b30-ff05-45fa-90db-5aa6bf4970ae" containerName="extract"
Jul 28 20:54:07.015069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014972    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed38cb3e-13af-4f80-b366-065ffe40d3f4" containerName="extract"
Jul 28 20:54:07.015069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014978    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="073b3df5-255a-43d8-bdf8-1e75b767507a" containerName="extract"
Jul 28 20:54:07.015069 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.014987    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4" containerName="extract"
Jul 28 20:54:07.018912 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.018894    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zq5m"
Jul 28 20:54:07.021377 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.021302    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-m9l2k\""
Jul 28 20:54:07.023494 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.021545    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 28 20:54:07.023494 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.021752    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 28 20:54:07.023494 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.021833    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 20:54:07.023494 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.021863    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 20:54:07.028912 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.028870    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-6zq5m"]
Jul 28 20:54:07.199394 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.199356    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f94621ee-e112-4507-b648-81c49a3d370b-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-6zq5m\" (UID: \"f94621ee-e112-4507-b648-81c49a3d370b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zq5m"
Jul 28 20:54:07.199550 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.199409    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f94621ee-e112-4507-b648-81c49a3d370b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-6zq5m\" (UID: \"f94621ee-e112-4507-b648-81c49a3d370b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zq5m"
Jul 28 20:54:07.199550 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.199467    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j26mz\" (UniqueName: \"kubernetes.io/projected/f94621ee-e112-4507-b648-81c49a3d370b-kube-api-access-j26mz\") pod \"kuadrant-console-plugin-66b5774985-6zq5m\" (UID: \"f94621ee-e112-4507-b648-81c49a3d370b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zq5m"
Jul 28 20:54:07.300315 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.300217    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j26mz\" (UniqueName: \"kubernetes.io/projected/f94621ee-e112-4507-b648-81c49a3d370b-kube-api-access-j26mz\") pod \"kuadrant-console-plugin-66b5774985-6zq5m\" (UID: \"f94621ee-e112-4507-b648-81c49a3d370b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zq5m"
Jul 28 20:54:07.300315 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.300271    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f94621ee-e112-4507-b648-81c49a3d370b-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-6zq5m\" (UID: \"f94621ee-e112-4507-b648-81c49a3d370b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zq5m"
Jul 28 20:54:07.300674 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.300330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f94621ee-e112-4507-b648-81c49a3d370b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-6zq5m\" (UID: \"f94621ee-e112-4507-b648-81c49a3d370b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zq5m"
Jul 28 20:54:07.300674 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:54:07.300457    2565 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 28 20:54:07.300674 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:54:07.300542    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f94621ee-e112-4507-b648-81c49a3d370b-plugin-serving-cert podName:f94621ee-e112-4507-b648-81c49a3d370b nodeName:}" failed. No retries permitted until 2026-07-28 20:54:07.800520949 +0000 UTC m=+403.024863914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/f94621ee-e112-4507-b648-81c49a3d370b-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-6zq5m" (UID: "f94621ee-e112-4507-b648-81c49a3d370b") : secret "plugin-serving-cert" not found
Jul 28 20:54:07.300943 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.300922    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f94621ee-e112-4507-b648-81c49a3d370b-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-6zq5m\" (UID: \"f94621ee-e112-4507-b648-81c49a3d370b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zq5m"
Jul 28 20:54:07.312234 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.312204    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j26mz\" (UniqueName: \"kubernetes.io/projected/f94621ee-e112-4507-b648-81c49a3d370b-kube-api-access-j26mz\") pod \"kuadrant-console-plugin-66b5774985-6zq5m\" (UID: \"f94621ee-e112-4507-b648-81c49a3d370b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zq5m"
Jul 28 20:54:07.804513 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.804470    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f94621ee-e112-4507-b648-81c49a3d370b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-6zq5m\" (UID: \"f94621ee-e112-4507-b648-81c49a3d370b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zq5m"
Jul 28 20:54:07.806900 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.806869    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f94621ee-e112-4507-b648-81c49a3d370b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-6zq5m\" (UID: \"f94621ee-e112-4507-b648-81c49a3d370b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zq5m"
Jul 28 20:54:07.933968 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:07.933930    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zq5m"
Jul 28 20:54:08.075080 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:08.075056    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-6zq5m"]
Jul 28 20:54:08.077134 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:54:08.077100    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf94621ee_e112_4507_b648_81c49a3d370b.slice/crio-d303bb65b8b4e9bf12aba6674ab19d6c8be1fa059035f9215c9458286d820343 WatchSource:0}: Error finding container d303bb65b8b4e9bf12aba6674ab19d6c8be1fa059035f9215c9458286d820343: Status 404 returned error can't find the container with id d303bb65b8b4e9bf12aba6674ab19d6c8be1fa059035f9215c9458286d820343
Jul 28 20:54:08.199173 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:08.199135    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zq5m" event={"ID":"f94621ee-e112-4507-b648-81c49a3d370b","Type":"ContainerStarted","Data":"d303bb65b8b4e9bf12aba6674ab19d6c8be1fa059035f9215c9458286d820343"}
Jul 28 20:54:29.194743 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.194703    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8666db5475-r86qc"]
Jul 28 20:54:29.198377 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.198360    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.208029 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.208005    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8666db5475-r86qc"]
Jul 28 20:54:29.291013 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.290978    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/769c953d-69e2-4ccd-9017-ec62469eb8aa-console-config\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.291013 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.291018    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/769c953d-69e2-4ccd-9017-ec62469eb8aa-console-oauth-config\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.291256 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.291043    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/769c953d-69e2-4ccd-9017-ec62469eb8aa-service-ca\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.291256 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.291133    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/769c953d-69e2-4ccd-9017-ec62469eb8aa-console-serving-cert\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.291256 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.291176    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/769c953d-69e2-4ccd-9017-ec62469eb8aa-trusted-ca-bundle\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.291256 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.291193    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkmh2\" (UniqueName: \"kubernetes.io/projected/769c953d-69e2-4ccd-9017-ec62469eb8aa-kube-api-access-dkmh2\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.291410 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.291275    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/769c953d-69e2-4ccd-9017-ec62469eb8aa-oauth-serving-cert\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.331230 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.331197    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zq5m" event={"ID":"f94621ee-e112-4507-b648-81c49a3d370b","Type":"ContainerStarted","Data":"efde44d90e195e661070d743533319ad1f950a6bb3f43ebb5fd0a063036bb7ee"}
Jul 28 20:54:29.341262 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.341217    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6zq5m" podStartSLOduration=3.162018572 podStartE2EDuration="23.341202816s" totalImagesPullingTime="20.179184244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:54:29.340192051 +0000 UTC m=+424.564535037" watchObservedRunningTime="2026-07-28 20:54:29.341202816 +0000 UTC m=+424.565545802"
Jul 28 20:54:29.392312 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.392274    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/769c953d-69e2-4ccd-9017-ec62469eb8aa-trusted-ca-bundle\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.392482 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.392326    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkmh2\" (UniqueName: \"kubernetes.io/projected/769c953d-69e2-4ccd-9017-ec62469eb8aa-kube-api-access-dkmh2\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.392482 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.392400    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/769c953d-69e2-4ccd-9017-ec62469eb8aa-oauth-serving-cert\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.392596 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.392479    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/769c953d-69e2-4ccd-9017-ec62469eb8aa-console-config\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.392596 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.392511    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/769c953d-69e2-4ccd-9017-ec62469eb8aa-console-oauth-config\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.392829 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.392787    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/769c953d-69e2-4ccd-9017-ec62469eb8aa-service-ca\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.392922 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.392884    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/769c953d-69e2-4ccd-9017-ec62469eb8aa-console-serving-cert\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.393214 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.393192    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/769c953d-69e2-4ccd-9017-ec62469eb8aa-oauth-serving-cert\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.393462 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.393440    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/769c953d-69e2-4ccd-9017-ec62469eb8aa-service-ca\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.393519 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.393440    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/769c953d-69e2-4ccd-9017-ec62469eb8aa-console-config\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.393573 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.393549    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/769c953d-69e2-4ccd-9017-ec62469eb8aa-trusted-ca-bundle\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.395283 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.395262    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/769c953d-69e2-4ccd-9017-ec62469eb8aa-console-serving-cert\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.395346 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.395262    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/769c953d-69e2-4ccd-9017-ec62469eb8aa-console-oauth-config\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.404503 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.404484    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkmh2\" (UniqueName: \"kubernetes.io/projected/769c953d-69e2-4ccd-9017-ec62469eb8aa-kube-api-access-dkmh2\") pod \"console-8666db5475-r86qc\" (UID: \"769c953d-69e2-4ccd-9017-ec62469eb8aa\") " pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.507960 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.507866    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:29.660329 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:29.660302    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8666db5475-r86qc"]
Jul 28 20:54:29.662082 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:54:29.662057    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769c953d_69e2_4ccd_9017_ec62469eb8aa.slice/crio-9c1a71ac2d4d92ae3c3ea0bc8c752a8406e3839516c5c1f5edcf0faf606622b2 WatchSource:0}: Error finding container 9c1a71ac2d4d92ae3c3ea0bc8c752a8406e3839516c5c1f5edcf0faf606622b2: Status 404 returned error can't find the container with id 9c1a71ac2d4d92ae3c3ea0bc8c752a8406e3839516c5c1f5edcf0faf606622b2
Jul 28 20:54:30.331761 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:30.331728    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8666db5475-r86qc" event={"ID":"769c953d-69e2-4ccd-9017-ec62469eb8aa","Type":"ContainerStarted","Data":"057d66f822ac36ac8d9b706ba84e4bb4ade222f62d3ab22b11c1c8808c9132b7"}
Jul 28 20:54:30.332212 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:30.331778    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8666db5475-r86qc" event={"ID":"769c953d-69e2-4ccd-9017-ec62469eb8aa","Type":"ContainerStarted","Data":"9c1a71ac2d4d92ae3c3ea0bc8c752a8406e3839516c5c1f5edcf0faf606622b2"}
Jul 28 20:54:30.352439 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:30.352395    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8666db5475-r86qc" podStartSLOduration=1.352381073 podStartE2EDuration="1.352381073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:54:30.349942282 +0000 UTC m=+425.574285268" watchObservedRunningTime="2026-07-28 20:54:30.352381073 +0000 UTC m=+425.576724056"
Jul 28 20:54:39.508006 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:39.507959    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:39.508403 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:39.508275    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:39.512703 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:39.512683    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:40.382771 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:40.382740    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8666db5475-r86qc"
Jul 28 20:54:40.428563 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:54:40.428530    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78659d4698-4w96c"]
Jul 28 20:55:05.450373 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.450306    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78659d4698-4w96c" podUID="68bfe83f-1cf6-460f-9660-c170a802e1c2" containerName="console" containerID="cri-o://3ebeedd8c1096cee7fb48b3237686d7b25b0506abce2a450b27bf63cd7c9ac86" gracePeriod=15
Jul 28 20:55:05.708193 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.708125    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:55:05.722857 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.722827    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68bfe83f-1cf6-460f-9660-c170a802e1c2-console-serving-cert\") pod \"68bfe83f-1cf6-460f-9660-c170a802e1c2\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") "
Jul 28 20:55:05.723002 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.722868    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-oauth-serving-cert\") pod \"68bfe83f-1cf6-460f-9660-c170a802e1c2\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") "
Jul 28 20:55:05.723002 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.722916    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-service-ca\") pod \"68bfe83f-1cf6-460f-9660-c170a802e1c2\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") "
Jul 28 20:55:05.723090 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.723021    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-trusted-ca-bundle\") pod \"68bfe83f-1cf6-460f-9660-c170a802e1c2\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") "
Jul 28 20:55:05.723090 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.723076    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-console-config\") pod \"68bfe83f-1cf6-460f-9660-c170a802e1c2\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") "
Jul 28 20:55:05.723169 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.723119    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68bfe83f-1cf6-460f-9660-c170a802e1c2-console-oauth-config\") pod \"68bfe83f-1cf6-460f-9660-c170a802e1c2\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") "
Jul 28 20:55:05.723169 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.723151    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9pld\" (UniqueName: \"kubernetes.io/projected/68bfe83f-1cf6-460f-9660-c170a802e1c2-kube-api-access-c9pld\") pod \"68bfe83f-1cf6-460f-9660-c170a802e1c2\" (UID: \"68bfe83f-1cf6-460f-9660-c170a802e1c2\") "
Jul 28 20:55:05.723434 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.723395    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "68bfe83f-1cf6-460f-9660-c170a802e1c2" (UID: "68bfe83f-1cf6-460f-9660-c170a802e1c2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:55:05.723590 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.723468    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-service-ca" (OuterVolumeSpecName: "service-ca") pod "68bfe83f-1cf6-460f-9660-c170a802e1c2" (UID: "68bfe83f-1cf6-460f-9660-c170a802e1c2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:55:05.723590 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.723505    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-console-config" (OuterVolumeSpecName: "console-config") pod "68bfe83f-1cf6-460f-9660-c170a802e1c2" (UID: "68bfe83f-1cf6-460f-9660-c170a802e1c2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:55:05.723765 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.723600    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "68bfe83f-1cf6-460f-9660-c170a802e1c2" (UID: "68bfe83f-1cf6-460f-9660-c170a802e1c2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:55:05.725717 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.725616    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68bfe83f-1cf6-460f-9660-c170a802e1c2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "68bfe83f-1cf6-460f-9660-c170a802e1c2" (UID: "68bfe83f-1cf6-460f-9660-c170a802e1c2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:55:05.725717 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.725656    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68bfe83f-1cf6-460f-9660-c170a802e1c2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "68bfe83f-1cf6-460f-9660-c170a802e1c2" (UID: "68bfe83f-1cf6-460f-9660-c170a802e1c2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:55:05.726539 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.726512    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bfe83f-1cf6-460f-9660-c170a802e1c2-kube-api-access-c9pld" (OuterVolumeSpecName: "kube-api-access-c9pld") pod "68bfe83f-1cf6-460f-9660-c170a802e1c2" (UID: "68bfe83f-1cf6-460f-9660-c170a802e1c2"). InnerVolumeSpecName "kube-api-access-c9pld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:55:05.824424 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.824384    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:55:05.824424 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.824417    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-trusted-ca-bundle\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:55:05.824424 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.824428    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-console-config\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:55:05.824701 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.824437    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68bfe83f-1cf6-460f-9660-c170a802e1c2-console-oauth-config\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:55:05.824701 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.824446    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9pld\" (UniqueName: \"kubernetes.io/projected/68bfe83f-1cf6-460f-9660-c170a802e1c2-kube-api-access-c9pld\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:55:05.824701 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.824454    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68bfe83f-1cf6-460f-9660-c170a802e1c2-console-serving-cert\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:55:05.824701 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:05.824463    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68bfe83f-1cf6-460f-9660-c170a802e1c2-oauth-serving-cert\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 20:55:06.509147 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:06.509111    2565 generic.go:356] "Generic (PLEG): container finished" podID="68bfe83f-1cf6-460f-9660-c170a802e1c2" containerID="3ebeedd8c1096cee7fb48b3237686d7b25b0506abce2a450b27bf63cd7c9ac86" exitCode=0
Jul 28 20:55:06.509582 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:06.509157    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78659d4698-4w96c"
Jul 28 20:55:06.509582 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:06.509195    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78659d4698-4w96c" event={"ID":"68bfe83f-1cf6-460f-9660-c170a802e1c2","Type":"ContainerDied","Data":"3ebeedd8c1096cee7fb48b3237686d7b25b0506abce2a450b27bf63cd7c9ac86"}
Jul 28 20:55:06.509582 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:06.509250    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78659d4698-4w96c" event={"ID":"68bfe83f-1cf6-460f-9660-c170a802e1c2","Type":"ContainerDied","Data":"b5a0f1838914e7c418dd5e59402a682e6fa9e8df77d8fdedbdfe2928f8b69465"}
Jul 28 20:55:06.509582 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:06.509273    2565 scope.go:117] "RemoveContainer" containerID="3ebeedd8c1096cee7fb48b3237686d7b25b0506abce2a450b27bf63cd7c9ac86"
Jul 28 20:55:06.518380 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:06.518359    2565 scope.go:117] "RemoveContainer" containerID="3ebeedd8c1096cee7fb48b3237686d7b25b0506abce2a450b27bf63cd7c9ac86"
Jul 28 20:55:06.518691 ip-10-0-134-170 kubenswrapper[2565]: E0728 20:55:06.518665    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ebeedd8c1096cee7fb48b3237686d7b25b0506abce2a450b27bf63cd7c9ac86\": container with ID starting with 3ebeedd8c1096cee7fb48b3237686d7b25b0506abce2a450b27bf63cd7c9ac86 not found: ID does not exist" containerID="3ebeedd8c1096cee7fb48b3237686d7b25b0506abce2a450b27bf63cd7c9ac86"
Jul 28 20:55:06.518775 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:06.518700    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ebeedd8c1096cee7fb48b3237686d7b25b0506abce2a450b27bf63cd7c9ac86"} err="failed to get container status \"3ebeedd8c1096cee7fb48b3237686d7b25b0506abce2a450b27bf63cd7c9ac86\": rpc error: code = NotFound desc = could not find container \"3ebeedd8c1096cee7fb48b3237686d7b25b0506abce2a450b27bf63cd7c9ac86\": container with ID starting with 3ebeedd8c1096cee7fb48b3237686d7b25b0506abce2a450b27bf63cd7c9ac86 not found: ID does not exist"
Jul 28 20:55:06.529878 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:06.529851    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78659d4698-4w96c"]
Jul 28 20:55:06.532465 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:06.532436    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78659d4698-4w96c"]
Jul 28 20:55:07.286939 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:07.286899    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 20:55:07.287243 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:07.287231    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68bfe83f-1cf6-460f-9660-c170a802e1c2" containerName="console"
Jul 28 20:55:07.287337 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:07.287245    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="68bfe83f-1cf6-460f-9660-c170a802e1c2" containerName="console"
Jul 28 20:55:07.287337 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:07.287310    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="68bfe83f-1cf6-460f-9660-c170a802e1c2" containerName="console"
Jul 28 20:55:07.292007 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:07.291987    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"
Jul 28 20:55:07.293741 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:07.293717    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 28 20:55:07.298213 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:07.298186    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 20:55:07.331743 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:07.331708    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68bfe83f-1cf6-460f-9660-c170a802e1c2" path="/var/lib/kubelet/pods/68bfe83f-1cf6-460f-9660-c170a802e1c2/volumes"
Jul 28 20:55:07.331991 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:07.331978    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 20:55:07.335031 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:07.335009    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/37a1bb31-c8f7-465e-8610-e7ae408a4a74-config-file\") pod \"limitador-limitador-54d6d988d5-nbrxn\" (UID: \"37a1bb31-c8f7-465e-8610-e7ae408a4a74\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"
Jul 28 20:55:07.335091 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:07.335055    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svx8s\" (UniqueName: \"kubernetes.io/projected/37a1bb31-c8f7-465e-8610-e7ae408a4a74-kube-api-access-svx8s\") pod \"limitador-limitador-54d6d988d5-nbrxn\" (UID: \"37a1bb31-c8f7-465e-8610-e7ae408a4a74\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"
Jul 28 20:55:07.436230 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:07.436188    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/37a1bb31-c8f7-465e-8610-e7ae408a4a74-config-file\") pod \"limitador-limitador-54d6d988d5-nbrxn\" (UID: \"37a1bb31-c8f7-465e-8610-e7ae408a4a74\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"
Jul 28 20:55:07.436454 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:07.436247    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svx8s\" (UniqueName: \"kubernetes.io/projected/37a1bb31-c8f7-465e-8610-e7ae408a4a74-kube-api-access-svx8s\") pod \"limitador-limitador-54d6d988d5-nbrxn\" (UID: \"37a1bb31-c8f7-465e-8610-e7ae408a4a74\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"
Jul 28 20:55:07.436891 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:07.436869    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/37a1bb31-c8f7-465e-8610-e7ae408a4a74-config-file\") pod \"limitador-limitador-54d6d988d5-nbrxn\" (UID: \"37a1bb31-c8f7-465e-8610-e7ae408a4a74\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"
Jul 28 20:55:07.445521 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:07.445494    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svx8s\" (UniqueName: \"kubernetes.io/projected/37a1bb31-c8f7-465e-8610-e7ae408a4a74-kube-api-access-svx8s\") pod \"limitador-limitador-54d6d988d5-nbrxn\" (UID: \"37a1bb31-c8f7-465e-8610-e7ae408a4a74\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"
Jul 28 20:55:07.602963 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:07.602861    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"
Jul 28 20:55:07.757611 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:07.757581    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 20:55:07.760094 ip-10-0-134-170 kubenswrapper[2565]: W0728 20:55:07.760064    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a1bb31_c8f7_465e_8610_e7ae408a4a74.slice/crio-802798e7e49d09914e32f9c77b858c9b48e32a38d7b2194a344f0ef89d5d6084 WatchSource:0}: Error finding container 802798e7e49d09914e32f9c77b858c9b48e32a38d7b2194a344f0ef89d5d6084: Status 404 returned error can't find the container with id 802798e7e49d09914e32f9c77b858c9b48e32a38d7b2194a344f0ef89d5d6084
Jul 28 20:55:08.523103 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:08.523064    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-nbrxn" event={"ID":"37a1bb31-c8f7-465e-8610-e7ae408a4a74","Type":"ContainerStarted","Data":"802798e7e49d09914e32f9c77b858c9b48e32a38d7b2194a344f0ef89d5d6084"}
Jul 28 20:55:10.537460 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:10.537425    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-nbrxn" event={"ID":"37a1bb31-c8f7-465e-8610-e7ae408a4a74","Type":"ContainerStarted","Data":"69f75c753aa0206c33ae2e10099ffb76d1e1ed07d0610e759687a11c2132d144"}
Jul 28 20:55:10.537865 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:10.537497    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"
Jul 28 20:55:10.558787 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:10.558732    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-nbrxn" podStartSLOduration=1.1516852069999999 podStartE2EDuration="3.55871533s" totalImagesPullingTime="2.407030123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:55:10.552516981 +0000 UTC m=+465.776859966" watchObservedRunningTime="2026-07-28 20:55:10.55871533 +0000 UTC m=+465.783058380"
Jul 28 20:55:21.543657 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:55:21.543607    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"
Jul 28 20:56:16.516067 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:56:16.516031    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 20:56:33.451541 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:56:33.451464    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 20:56:39.651324 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:56:39.651285    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 20:56:47.064054 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:56:47.064018    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 20:57:04.458996 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:57:04.458959    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 20:57:13.953734 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:57:13.953698    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 20:57:33.068815 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:57:33.068775    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 20:58:47.956351 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:58:47.956312    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 20:58:56.863483 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:58:56.863446    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 20:59:07.760587 ip-10-0-134-170 kubenswrapper[2565]: I0728 20:59:07.760552    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:00:00.147936 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:00:00.147842    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754540-q2htm"]
Jul 28 21:00:00.151226 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:00:00.151209    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754540-q2htm"
Jul 28 21:00:00.153193 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:00:00.153173    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w22xz\""
Jul 28 21:00:00.158430 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:00:00.158405    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754540-q2htm"]
Jul 28 21:00:00.162905 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:00:00.162883    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754540-q2htm"
Jul 28 21:00:00.302692 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:00:00.302665    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754540-q2htm"]
Jul 28 21:00:00.304648 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:00:00.304605    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5cb520e_0e96_4ca8_893a_63cbcbda2ab8.slice/crio-c24ee43d5c292319d783ee90e87d1feccba5b56bdabc27d4ebe29d3ffc384706 WatchSource:0}: Error finding container c24ee43d5c292319d783ee90e87d1feccba5b56bdabc27d4ebe29d3ffc384706: Status 404 returned error can't find the container with id c24ee43d5c292319d783ee90e87d1feccba5b56bdabc27d4ebe29d3ffc384706
Jul 28 21:00:00.306729 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:00:00.306707    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 21:00:01.022162 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:00:01.022129    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754540-q2htm" event={"ID":"d5cb520e-0e96-4ca8-893a-63cbcbda2ab8","Type":"ContainerStarted","Data":"c24ee43d5c292319d783ee90e87d1feccba5b56bdabc27d4ebe29d3ffc384706"}
Jul 28 21:00:03.033894 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:00:03.033858    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754540-q2htm" event={"ID":"d5cb520e-0e96-4ca8-893a-63cbcbda2ab8","Type":"ContainerStarted","Data":"3471f84a585ca764356ed5f3e10b3590ac12f40d3fa7f5c6113a0e3ab6b9405e"}
Jul 28 21:00:03.048210 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:00:03.048159    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754540-q2htm" podStartSLOduration=0.924426833 podStartE2EDuration="3.048143182s" totalImagesPullingTime="2.123716349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:00:03.046277955 +0000 UTC m=+758.270620942" watchObservedRunningTime="2026-07-28 21:00:03.048143182 +0000 UTC m=+758.272486168"
Jul 28 21:00:04.040605 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:00:04.040569    2565 generic.go:356] "Generic (PLEG): container finished" podID="d5cb520e-0e96-4ca8-893a-63cbcbda2ab8" containerID="3471f84a585ca764356ed5f3e10b3590ac12f40d3fa7f5c6113a0e3ab6b9405e" exitCode=0
Jul 28 21:00:04.040999 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:00:04.040665    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754540-q2htm" event={"ID":"d5cb520e-0e96-4ca8-893a-63cbcbda2ab8","Type":"ContainerDied","Data":"3471f84a585ca764356ed5f3e10b3590ac12f40d3fa7f5c6113a0e3ab6b9405e"}
Jul 28 21:00:05.183361 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:00:05.183335    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754540-q2htm"
Jul 28 21:00:06.052157 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:00:06.052127    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754540-q2htm"
Jul 28 21:00:06.052157 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:00:06.052143    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754540-q2htm" event={"ID":"d5cb520e-0e96-4ca8-893a-63cbcbda2ab8","Type":"ContainerDied","Data":"c24ee43d5c292319d783ee90e87d1feccba5b56bdabc27d4ebe29d3ffc384706"}
Jul 28 21:00:06.052492 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:00:06.052198    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c24ee43d5c292319d783ee90e87d1feccba5b56bdabc27d4ebe29d3ffc384706"
Jul 28 21:00:50.063314 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:00:50.063227    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:01:01.459698 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:01:01.459647    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:01:36.857722 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:01:36.857686    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:01:46.249335 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:01:46.249296    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:01:54.957377 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:01:54.957340    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:02:04.150005 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:02:04.149961    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:02:14.060003 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:02:14.055684    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:02:22.961086 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:02:22.961009    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:02:35.261088 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:02:35.261047    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:02:45.356216 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:02:45.356161    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:03:07.448942 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:03:07.448904    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:03:20.768192 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:03:20.768149    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:03:41.652096 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:03:41.652059    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:03:51.563326 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:03:51.563243    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:04:01.356351 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:04:01.356307    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:04:18.062599 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:04:18.062563    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:04:28.960305 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:04:28.960270    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:04:38.770435 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:04:38.770395    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:04:47.550971 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:04:47.550934    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:04:56.749224 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:04:56.749187    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:06:46.060874 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:06:46.060839    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:06:56.962748 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:06:56.962662    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:07:13.459219 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:07:13.459178    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:07:28.948940 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:07:28.948905    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:07:41.755942 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:07:41.755897    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:07:49.947384 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:07:49.947340    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:07:59.959983 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:07:59.959946    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:08:11.459031 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:08:11.458995    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:08:20.152591 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:08:20.152505    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:08:30.960341 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:08:30.960291    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:09:07.651299 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:09:07.651267    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:09:17.362214 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:09:17.362180    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:09:35.558324 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:09:35.558284    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:09:54.058703 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:09:54.058596    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:10:31.860460 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:10:31.860423    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:10:44.660100 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:10:44.660059    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:10:53.859222 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:10:53.859187    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:11:03.754056 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:11:03.754016    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:11:16.356707 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:11:16.356666    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:11:24.954820 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:11:24.954737    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:11:35.060980 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:11:35.060944    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:11:44.462188 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:11:44.462152    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:11:53.656586 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:11:53.656552    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:12:06.853555 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:12:06.853516    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:12:16.194228 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:12:16.194196    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:12:25.651263 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:12:25.651221    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:12:34.652930 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:12:34.652894    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:12:47.455510 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:12:47.455424    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:12:55.269133 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:12:55.269090    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:13:07.263751 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:13:07.263716    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:13:17.860918 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:13:17.860880    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:13:31.453187 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:13:31.453150    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:13:44.165707 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:13:44.165667    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:13:56.762671 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:13:56.762615    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:14:05.662173 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.662131    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p"]
Jul 28 21:14:05.662692 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.662479    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5cb520e-0e96-4ca8-893a-63cbcbda2ab8" containerName="cleanup"
Jul 28 21:14:05.662692 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.662490    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5cb520e-0e96-4ca8-893a-63cbcbda2ab8" containerName="cleanup"
Jul 28 21:14:05.662692 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.662553    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5cb520e-0e96-4ca8-893a-63cbcbda2ab8" containerName="cleanup"
Jul 28 21:14:05.665588 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.665572    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p"
Jul 28 21:14:05.667401 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.667378    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a4f1163e\""
Jul 28 21:14:05.667500 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.667427    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a4f1163e-dockercfg-mkc8c\""
Jul 28 21:14:05.675533 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.675510    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p"]
Jul 28 21:14:05.700072 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.700044    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk"]
Jul 28 21:14:05.703680 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.703656    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk"
Jul 28 21:14:05.713508 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.713482    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk"]
Jul 28 21:14:05.750764 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.750731    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/627829c7-a58f-4829-a90e-0ce3c6f8651b-plugins-config-volume\") pod \"payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk\" (UID: \"627829c7-a58f-4829-a90e-0ce3c6f8651b\") " pod="openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk"
Jul 28 21:14:05.750918 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.750783    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prdvt\" (UniqueName: \"kubernetes.io/projected/627829c7-a58f-4829-a90e-0ce3c6f8651b-kube-api-access-prdvt\") pod \"payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk\" (UID: \"627829c7-a58f-4829-a90e-0ce3c6f8651b\") " pod="openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk"
Jul 28 21:14:05.851378 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.851341    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b857bf-b1e4-419f-845f-3b600375ad19-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p\" (UID: \"36b857bf-b1e4-419f-845f-3b600375ad19\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p"
Jul 28 21:14:05.851545 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.851387    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpw5s\" (UniqueName: \"kubernetes.io/projected/36b857bf-b1e4-419f-845f-3b600375ad19-kube-api-access-tpw5s\") pod \"payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p\" (UID: \"36b857bf-b1e4-419f-845f-3b600375ad19\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p"
Jul 28 21:14:05.851545 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.851473    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/627829c7-a58f-4829-a90e-0ce3c6f8651b-plugins-config-volume\") pod \"payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk\" (UID: \"627829c7-a58f-4829-a90e-0ce3c6f8651b\") " pod="openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk"
Jul 28 21:14:05.851545 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.851539    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prdvt\" (UniqueName: \"kubernetes.io/projected/627829c7-a58f-4829-a90e-0ce3c6f8651b-kube-api-access-prdvt\") pod \"payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk\" (UID: \"627829c7-a58f-4829-a90e-0ce3c6f8651b\") " pod="openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk"
Jul 28 21:14:05.852200 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.852181    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/627829c7-a58f-4829-a90e-0ce3c6f8651b-plugins-config-volume\") pod \"payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk\" (UID: \"627829c7-a58f-4829-a90e-0ce3c6f8651b\") " pod="openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk"
Jul 28 21:14:05.859234 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.859208    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prdvt\" (UniqueName: \"kubernetes.io/projected/627829c7-a58f-4829-a90e-0ce3c6f8651b-kube-api-access-prdvt\") pod \"payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk\" (UID: \"627829c7-a58f-4829-a90e-0ce3c6f8651b\") " pod="openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk"
Jul 28 21:14:05.952401 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.952369    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b857bf-b1e4-419f-845f-3b600375ad19-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p\" (UID: \"36b857bf-b1e4-419f-845f-3b600375ad19\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p"
Jul 28 21:14:05.952550 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.952407    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpw5s\" (UniqueName: \"kubernetes.io/projected/36b857bf-b1e4-419f-845f-3b600375ad19-kube-api-access-tpw5s\") pod \"payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p\" (UID: \"36b857bf-b1e4-419f-845f-3b600375ad19\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p"
Jul 28 21:14:05.953033 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.953013    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b857bf-b1e4-419f-845f-3b600375ad19-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p\" (UID: \"36b857bf-b1e4-419f-845f-3b600375ad19\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p"
Jul 28 21:14:05.959893 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.959866    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpw5s\" (UniqueName: \"kubernetes.io/projected/36b857bf-b1e4-419f-845f-3b600375ad19-kube-api-access-tpw5s\") pod \"payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p\" (UID: \"36b857bf-b1e4-419f-845f-3b600375ad19\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p"
Jul 28 21:14:05.976567 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:05.976541    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p"
Jul 28 21:14:06.016739 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:06.016693    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk"
Jul 28 21:14:06.124292 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:06.124256    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p"]
Jul 28 21:14:06.124887 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:14:06.124859    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b857bf_b1e4_419f_845f_3b600375ad19.slice/crio-f56ff3fa1f5f1470f03ec2785fc619ac90c25ac563ecf2eae5c84a9edec0bcfa WatchSource:0}: Error finding container f56ff3fa1f5f1470f03ec2785fc619ac90c25ac563ecf2eae5c84a9edec0bcfa: Status 404 returned error can't find the container with id f56ff3fa1f5f1470f03ec2785fc619ac90c25ac563ecf2eae5c84a9edec0bcfa
Jul 28 21:14:06.127150 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:06.127134    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 21:14:06.170703 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:06.170611    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk"]
Jul 28 21:14:06.174104 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:14:06.174077    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627829c7_a58f_4829_a90e_0ce3c6f8651b.slice/crio-9bff8e70609d42681dfe1aa45db8f00b3e8d5ce767aef319d0a34bf6611b5533 WatchSource:0}: Error finding container 9bff8e70609d42681dfe1aa45db8f00b3e8d5ce767aef319d0a34bf6611b5533: Status 404 returned error can't find the container with id 9bff8e70609d42681dfe1aa45db8f00b3e8d5ce767aef319d0a34bf6611b5533
Jul 28 21:14:06.297728 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:06.297614    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p" event={"ID":"36b857bf-b1e4-419f-845f-3b600375ad19","Type":"ContainerStarted","Data":"f56ff3fa1f5f1470f03ec2785fc619ac90c25ac563ecf2eae5c84a9edec0bcfa"}
Jul 28 21:14:06.298720 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:06.298697    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk" event={"ID":"627829c7-a58f-4829-a90e-0ce3c6f8651b","Type":"ContainerStarted","Data":"9bff8e70609d42681dfe1aa45db8f00b3e8d5ce767aef319d0a34bf6611b5533"}
Jul 28 21:14:07.758929 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:07.758890    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:14:08.316462 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:08.316425    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk" event={"ID":"627829c7-a58f-4829-a90e-0ce3c6f8651b","Type":"ContainerStarted","Data":"24cd2128b75d4acb762e31d703cd1452793a9058db1b8022a40b29f5e338bf44"}
Jul 28 21:14:08.316708 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:08.316491    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk"
Jul 28 21:14:08.317904 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:08.317882    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p" event={"ID":"36b857bf-b1e4-419f-845f-3b600375ad19","Type":"ContainerStarted","Data":"055ab138b9575662054c5af9ae1f707bca9015aef30f6c0cec81020ad2205653"}
Jul 28 21:14:08.318015 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:08.317923    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p"
Jul 28 21:14:08.335278 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:08.335222    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk" podStartSLOduration=1.593757524 podStartE2EDuration="3.335201987s" totalImagesPullingTime="1.741444463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:14:08.332948585 +0000 UTC m=+1603.557291572" watchObservedRunningTime="2026-07-28 21:14:08.335201987 +0000 UTC m=+1603.559544974"
Jul 28 21:14:08.347117 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:08.347068    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p" podStartSLOduration=1.563677547 podStartE2EDuration="3.347051507s" totalImagesPullingTime="1.78337396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:14:08.345892752 +0000 UTC m=+1603.570235737" watchObservedRunningTime="2026-07-28 21:14:08.347051507 +0000 UTC m=+1603.571394492"
Jul 28 21:14:19.325058 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:19.324978    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p"
Jul 28 21:14:19.325058 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:19.325044    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk"
Jul 28 21:14:30.396271 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:30.396232    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1163e-sghfq"]
Jul 28 21:14:30.399674 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:30.399655    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1163e-sghfq"
Jul 28 21:14:30.401513 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:30.401491    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 21:14:30.401662 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:30.401491    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w22xz\""
Jul 28 21:14:30.406695 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:30.406667    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1163e-sghfq"]
Jul 28 21:14:30.459055 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:30.459016    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1939b336-b94a-43f4-ad17-a969ec15fa5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4f1163e-sghfq\" (UID: \"1939b336-b94a-43f4-ad17-a969ec15fa5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1163e-sghfq"
Jul 28 21:14:30.559841 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:30.559797    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1939b336-b94a-43f4-ad17-a969ec15fa5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4f1163e-sghfq\" (UID: \"1939b336-b94a-43f4-ad17-a969ec15fa5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1163e-sghfq"
Jul 28 21:14:30.560408 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:30.560387    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1939b336-b94a-43f4-ad17-a969ec15fa5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4f1163e-sghfq\" (UID: \"1939b336-b94a-43f4-ad17-a969ec15fa5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1163e-sghfq"
Jul 28 21:14:30.712455 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:30.712423    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1163e-sghfq"
Jul 28 21:14:30.844508 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:30.844477    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1163e-sghfq"]
Jul 28 21:14:30.846414 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:14:30.846387    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1939b336_b94a_43f4_ad17_a969ec15fa5d.slice/crio-cb5e998c894f7c66a2de42d2d65fc84cc6abaaf588dc04c5065f17a61ca885f9 WatchSource:0}: Error finding container cb5e998c894f7c66a2de42d2d65fc84cc6abaaf588dc04c5065f17a61ca885f9: Status 404 returned error can't find the container with id cb5e998c894f7c66a2de42d2d65fc84cc6abaaf588dc04c5065f17a61ca885f9
Jul 28 21:14:31.440767 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:31.440727    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1163e-sghfq" event={"ID":"1939b336-b94a-43f4-ad17-a969ec15fa5d","Type":"ContainerStarted","Data":"fc10835114614445c4d057ba4fdda061b27f0fc8273957c80b7e73eaa71cb0b4"}
Jul 28 21:14:31.441184 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:31.440786    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1163e-sghfq" event={"ID":"1939b336-b94a-43f4-ad17-a969ec15fa5d","Type":"ContainerStarted","Data":"cb5e998c894f7c66a2de42d2d65fc84cc6abaaf588dc04c5065f17a61ca885f9"}
Jul 28 21:14:31.455208 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:31.455164    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1163e-sghfq" podStartSLOduration=1.455147516 podStartE2EDuration="1.455147516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:14:31.453763048 +0000 UTC m=+1626.678106037" watchObservedRunningTime="2026-07-28 21:14:31.455147516 +0000 UTC m=+1626.679490504"
Jul 28 21:14:32.447563 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:32.447528    2565 generic.go:356] "Generic (PLEG): container finished" podID="1939b336-b94a-43f4-ad17-a969ec15fa5d" containerID="fc10835114614445c4d057ba4fdda061b27f0fc8273957c80b7e73eaa71cb0b4" exitCode=0
Jul 28 21:14:32.447978 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:32.447617    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1163e-sghfq" event={"ID":"1939b336-b94a-43f4-ad17-a969ec15fa5d","Type":"ContainerDied","Data":"fc10835114614445c4d057ba4fdda061b27f0fc8273957c80b7e73eaa71cb0b4"}
Jul 28 21:14:33.582356 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:33.582329    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1163e-sghfq"
Jul 28 21:14:33.689486 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:33.689457    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1939b336-b94a-43f4-ad17-a969ec15fa5d-maas-api-service-ca\") pod \"1939b336-b94a-43f4-ad17-a969ec15fa5d\" (UID: \"1939b336-b94a-43f4-ad17-a969ec15fa5d\") "
Jul 28 21:14:33.689866 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:33.689845    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1939b336-b94a-43f4-ad17-a969ec15fa5d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1939b336-b94a-43f4-ad17-a969ec15fa5d" (UID: "1939b336-b94a-43f4-ad17-a969ec15fa5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:14:33.791064 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:33.790974    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1939b336-b94a-43f4-ad17-a969ec15fa5d-maas-api-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:14:34.459033 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:34.458999    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1163e-sghfq"
Jul 28 21:14:34.459231 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:34.459001    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1163e-sghfq" event={"ID":"1939b336-b94a-43f4-ad17-a969ec15fa5d","Type":"ContainerDied","Data":"cb5e998c894f7c66a2de42d2d65fc84cc6abaaf588dc04c5065f17a61ca885f9"}
Jul 28 21:14:34.459231 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:34.459124    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb5e998c894f7c66a2de42d2d65fc84cc6abaaf588dc04c5065f17a61ca885f9"
Jul 28 21:14:35.684883 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:35.684850    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk"]
Jul 28 21:14:35.685313 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:35.685103    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk" podUID="627829c7-a58f-4829-a90e-0ce3c6f8651b" containerName="payload-processing" containerID="cri-o://24cd2128b75d4acb762e31d703cd1452793a9058db1b8022a40b29f5e338bf44" gracePeriod=30
Jul 28 21:14:35.693565 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:35.693399    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p"]
Jul 28 21:14:35.693736 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:35.693685    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p" podUID="36b857bf-b1e4-419f-845f-3b600375ad19" containerName="payload-pre-processing" containerID="cri-o://055ab138b9575662054c5af9ae1f707bca9015aef30f6c0cec81020ad2205653" gracePeriod=30
Jul 28 21:14:35.821365 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:35.821319    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1163e-sghfq"]
Jul 28 21:14:35.823755 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:35.823730    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1163e-sghfq"]
Jul 28 21:14:35.967107 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:35.967080    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk"
Jul 28 21:14:35.971535 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:35.971517    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p"
Jul 28 21:14:36.112986 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.112935    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpw5s\" (UniqueName: \"kubernetes.io/projected/36b857bf-b1e4-419f-845f-3b600375ad19-kube-api-access-tpw5s\") pod \"36b857bf-b1e4-419f-845f-3b600375ad19\" (UID: \"36b857bf-b1e4-419f-845f-3b600375ad19\") "
Jul 28 21:14:36.113186 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.113012    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b857bf-b1e4-419f-845f-3b600375ad19-plugins-config-volume\") pod \"36b857bf-b1e4-419f-845f-3b600375ad19\" (UID: \"36b857bf-b1e4-419f-845f-3b600375ad19\") "
Jul 28 21:14:36.113186 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.113048    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/627829c7-a58f-4829-a90e-0ce3c6f8651b-plugins-config-volume\") pod \"627829c7-a58f-4829-a90e-0ce3c6f8651b\" (UID: \"627829c7-a58f-4829-a90e-0ce3c6f8651b\") "
Jul 28 21:14:36.113186 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.113130    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prdvt\" (UniqueName: \"kubernetes.io/projected/627829c7-a58f-4829-a90e-0ce3c6f8651b-kube-api-access-prdvt\") pod \"627829c7-a58f-4829-a90e-0ce3c6f8651b\" (UID: \"627829c7-a58f-4829-a90e-0ce3c6f8651b\") "
Jul 28 21:14:36.113433 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.113392    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b857bf-b1e4-419f-845f-3b600375ad19-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "36b857bf-b1e4-419f-845f-3b600375ad19" (UID: "36b857bf-b1e4-419f-845f-3b600375ad19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:14:36.113492 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.113459    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/627829c7-a58f-4829-a90e-0ce3c6f8651b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "627829c7-a58f-4829-a90e-0ce3c6f8651b" (UID: "627829c7-a58f-4829-a90e-0ce3c6f8651b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:14:36.115332 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.115311    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b857bf-b1e4-419f-845f-3b600375ad19-kube-api-access-tpw5s" (OuterVolumeSpecName: "kube-api-access-tpw5s") pod "36b857bf-b1e4-419f-845f-3b600375ad19" (UID: "36b857bf-b1e4-419f-845f-3b600375ad19"). InnerVolumeSpecName "kube-api-access-tpw5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:14:36.115332 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.115318    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/627829c7-a58f-4829-a90e-0ce3c6f8651b-kube-api-access-prdvt" (OuterVolumeSpecName: "kube-api-access-prdvt") pod "627829c7-a58f-4829-a90e-0ce3c6f8651b" (UID: "627829c7-a58f-4829-a90e-0ce3c6f8651b"). InnerVolumeSpecName "kube-api-access-prdvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:14:36.214261 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.214175    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpw5s\" (UniqueName: \"kubernetes.io/projected/36b857bf-b1e4-419f-845f-3b600375ad19-kube-api-access-tpw5s\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:14:36.214261 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.214205    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b857bf-b1e4-419f-845f-3b600375ad19-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:14:36.214261 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.214215    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/627829c7-a58f-4829-a90e-0ce3c6f8651b-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:14:36.214261 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.214224    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prdvt\" (UniqueName: \"kubernetes.io/projected/627829c7-a58f-4829-a90e-0ce3c6f8651b-kube-api-access-prdvt\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:14:36.471153 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.471066    2565 generic.go:356] "Generic (PLEG): container finished" podID="627829c7-a58f-4829-a90e-0ce3c6f8651b" containerID="24cd2128b75d4acb762e31d703cd1452793a9058db1b8022a40b29f5e338bf44" exitCode=0
Jul 28 21:14:36.471153 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.471109    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk"
Jul 28 21:14:36.471379 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.471149    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk" event={"ID":"627829c7-a58f-4829-a90e-0ce3c6f8651b","Type":"ContainerDied","Data":"24cd2128b75d4acb762e31d703cd1452793a9058db1b8022a40b29f5e338bf44"}
Jul 28 21:14:36.471379 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.471225    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk" event={"ID":"627829c7-a58f-4829-a90e-0ce3c6f8651b","Type":"ContainerDied","Data":"9bff8e70609d42681dfe1aa45db8f00b3e8d5ce767aef319d0a34bf6611b5533"}
Jul 28 21:14:36.471379 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.471257    2565 scope.go:117] "RemoveContainer" containerID="24cd2128b75d4acb762e31d703cd1452793a9058db1b8022a40b29f5e338bf44"
Jul 28 21:14:36.472465 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.472441    2565 generic.go:356] "Generic (PLEG): container finished" podID="36b857bf-b1e4-419f-845f-3b600375ad19" containerID="055ab138b9575662054c5af9ae1f707bca9015aef30f6c0cec81020ad2205653" exitCode=0
Jul 28 21:14:36.472559 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.472487    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p"
Jul 28 21:14:36.472559 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.472524    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p" event={"ID":"36b857bf-b1e4-419f-845f-3b600375ad19","Type":"ContainerDied","Data":"055ab138b9575662054c5af9ae1f707bca9015aef30f6c0cec81020ad2205653"}
Jul 28 21:14:36.472679 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.472577    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p" event={"ID":"36b857bf-b1e4-419f-845f-3b600375ad19","Type":"ContainerDied","Data":"f56ff3fa1f5f1470f03ec2785fc619ac90c25ac563ecf2eae5c84a9edec0bcfa"}
Jul 28 21:14:36.481142 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.481039    2565 scope.go:117] "RemoveContainer" containerID="24cd2128b75d4acb762e31d703cd1452793a9058db1b8022a40b29f5e338bf44"
Jul 28 21:14:36.481369 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:14:36.481344    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24cd2128b75d4acb762e31d703cd1452793a9058db1b8022a40b29f5e338bf44\": container with ID starting with 24cd2128b75d4acb762e31d703cd1452793a9058db1b8022a40b29f5e338bf44 not found: ID does not exist" containerID="24cd2128b75d4acb762e31d703cd1452793a9058db1b8022a40b29f5e338bf44"
Jul 28 21:14:36.481480 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.481396    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24cd2128b75d4acb762e31d703cd1452793a9058db1b8022a40b29f5e338bf44"} err="failed to get container status \"24cd2128b75d4acb762e31d703cd1452793a9058db1b8022a40b29f5e338bf44\": rpc error: code = NotFound desc = could not find container \"24cd2128b75d4acb762e31d703cd1452793a9058db1b8022a40b29f5e338bf44\": container with ID starting with 24cd2128b75d4acb762e31d703cd1452793a9058db1b8022a40b29f5e338bf44 not found: ID does not exist"
Jul 28 21:14:36.481480 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.481418    2565 scope.go:117] "RemoveContainer" containerID="055ab138b9575662054c5af9ae1f707bca9015aef30f6c0cec81020ad2205653"
Jul 28 21:14:36.489493 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.489476    2565 scope.go:117] "RemoveContainer" containerID="055ab138b9575662054c5af9ae1f707bca9015aef30f6c0cec81020ad2205653"
Jul 28 21:14:36.489809 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:14:36.489790    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"055ab138b9575662054c5af9ae1f707bca9015aef30f6c0cec81020ad2205653\": container with ID starting with 055ab138b9575662054c5af9ae1f707bca9015aef30f6c0cec81020ad2205653 not found: ID does not exist" containerID="055ab138b9575662054c5af9ae1f707bca9015aef30f6c0cec81020ad2205653"
Jul 28 21:14:36.489870 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.489816    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055ab138b9575662054c5af9ae1f707bca9015aef30f6c0cec81020ad2205653"} err="failed to get container status \"055ab138b9575662054c5af9ae1f707bca9015aef30f6c0cec81020ad2205653\": rpc error: code = NotFound desc = could not find container \"055ab138b9575662054c5af9ae1f707bca9015aef30f6c0cec81020ad2205653\": container with ID starting with 055ab138b9575662054c5af9ae1f707bca9015aef30f6c0cec81020ad2205653 not found: ID does not exist"
Jul 28 21:14:36.500107 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.500079    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk"]
Jul 28 21:14:36.505981 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.505957    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a4f1163e-9fff87c4-jbljk"]
Jul 28 21:14:36.514189 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.514159    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p"]
Jul 28 21:14:36.516113 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:36.516093    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a4f1163e-76b4978c56-rbw9p"]
Jul 28 21:14:37.332264 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:37.332235    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1939b336-b94a-43f4-ad17-a969ec15fa5d" path="/var/lib/kubelet/pods/1939b336-b94a-43f4-ad17-a969ec15fa5d/volumes"
Jul 28 21:14:37.332797 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:37.332579    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36b857bf-b1e4-419f-845f-3b600375ad19" path="/var/lib/kubelet/pods/36b857bf-b1e4-419f-845f-3b600375ad19/volumes"
Jul 28 21:14:37.332905 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:37.332891    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="627829c7-a58f-4829-a90e-0ce3c6f8651b" path="/var/lib/kubelet/pods/627829c7-a58f-4829-a90e-0ce3c6f8651b/volumes"
Jul 28 21:14:39.455301 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:39.455260    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:14:40.364601 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.364568    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2"]
Jul 28 21:14:40.364925 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.364911    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36b857bf-b1e4-419f-845f-3b600375ad19" containerName="payload-pre-processing"
Jul 28 21:14:40.364972 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.364926    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="36b857bf-b1e4-419f-845f-3b600375ad19" containerName="payload-pre-processing"
Jul 28 21:14:40.364972 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.364940    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1939b336-b94a-43f4-ad17-a969ec15fa5d" containerName="revoke-keys"
Jul 28 21:14:40.364972 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.364945    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1939b336-b94a-43f4-ad17-a969ec15fa5d" containerName="revoke-keys"
Jul 28 21:14:40.364972 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.364956    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="627829c7-a58f-4829-a90e-0ce3c6f8651b" containerName="payload-processing"
Jul 28 21:14:40.364972 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.364962    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="627829c7-a58f-4829-a90e-0ce3c6f8651b" containerName="payload-processing"
Jul 28 21:14:40.365122 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.365018    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="36b857bf-b1e4-419f-845f-3b600375ad19" containerName="payload-pre-processing"
Jul 28 21:14:40.365122 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.365027    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1939b336-b94a-43f4-ad17-a969ec15fa5d" containerName="revoke-keys"
Jul 28 21:14:40.365122 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.365033    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="627829c7-a58f-4829-a90e-0ce3c6f8651b" containerName="payload-processing"
Jul 28 21:14:40.369724 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.369705    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2"
Jul 28 21:14:40.371537 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.371514    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-3ba07b96\""
Jul 28 21:14:40.378263 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.378241    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2"]
Jul 28 21:14:40.451302 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.451264    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr7z7\" (UniqueName: \"kubernetes.io/projected/77100a69-c174-4476-b7d2-32fd0b424d3c-kube-api-access-wr7z7\") pod \"payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2\" (UID: \"77100a69-c174-4476-b7d2-32fd0b424d3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2"
Jul 28 21:14:40.451480 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.451316    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77100a69-c174-4476-b7d2-32fd0b424d3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2\" (UID: \"77100a69-c174-4476-b7d2-32fd0b424d3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2"
Jul 28 21:14:40.487875 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.487842    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw"]
Jul 28 21:14:40.491363 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.491342    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw"
Jul 28 21:14:40.492905 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.492885    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-3ba07b96-dockercfg-gff5l\""
Jul 28 21:14:40.501721 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.501692    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw"]
Jul 28 21:14:40.552048 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.552014    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr7z7\" (UniqueName: \"kubernetes.io/projected/77100a69-c174-4476-b7d2-32fd0b424d3c-kube-api-access-wr7z7\") pod \"payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2\" (UID: \"77100a69-c174-4476-b7d2-32fd0b424d3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2"
Jul 28 21:14:40.552217 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.552063    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77100a69-c174-4476-b7d2-32fd0b424d3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2\" (UID: \"77100a69-c174-4476-b7d2-32fd0b424d3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2"
Jul 28 21:14:40.552837 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.552817    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77100a69-c174-4476-b7d2-32fd0b424d3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2\" (UID: \"77100a69-c174-4476-b7d2-32fd0b424d3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2"
Jul 28 21:14:40.558927 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.558904    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr7z7\" (UniqueName: \"kubernetes.io/projected/77100a69-c174-4476-b7d2-32fd0b424d3c-kube-api-access-wr7z7\") pod \"payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2\" (UID: \"77100a69-c174-4476-b7d2-32fd0b424d3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2"
Jul 28 21:14:40.652717 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.652613    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/351892c8-68a8-4218-b95e-d20be76ab83b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw\" (UID: \"351892c8-68a8-4218-b95e-d20be76ab83b\") " pod="openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw"
Jul 28 21:14:40.652717 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.652686    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdrjs\" (UniqueName: \"kubernetes.io/projected/351892c8-68a8-4218-b95e-d20be76ab83b-kube-api-access-zdrjs\") pod \"payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw\" (UID: \"351892c8-68a8-4218-b95e-d20be76ab83b\") " pod="openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw"
Jul 28 21:14:40.680758 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.680731    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2"
Jul 28 21:14:40.753988 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.753948    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/351892c8-68a8-4218-b95e-d20be76ab83b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw\" (UID: \"351892c8-68a8-4218-b95e-d20be76ab83b\") " pod="openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw"
Jul 28 21:14:40.754155 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.754019    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdrjs\" (UniqueName: \"kubernetes.io/projected/351892c8-68a8-4218-b95e-d20be76ab83b-kube-api-access-zdrjs\") pod \"payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw\" (UID: \"351892c8-68a8-4218-b95e-d20be76ab83b\") " pod="openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw"
Jul 28 21:14:40.754704 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.754687    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/351892c8-68a8-4218-b95e-d20be76ab83b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw\" (UID: \"351892c8-68a8-4218-b95e-d20be76ab83b\") " pod="openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw"
Jul 28 21:14:40.761291 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.761257    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdrjs\" (UniqueName: \"kubernetes.io/projected/351892c8-68a8-4218-b95e-d20be76ab83b-kube-api-access-zdrjs\") pod \"payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw\" (UID: \"351892c8-68a8-4218-b95e-d20be76ab83b\") " pod="openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw"
Jul 28 21:14:40.802853 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.802825    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw"
Jul 28 21:14:40.812698 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.812677    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2"]
Jul 28 21:14:40.815823 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:14:40.815798    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77100a69_c174_4476_b7d2_32fd0b424d3c.slice/crio-401c7a168ef49fb893c137be8ecad1c0b17646efeddb5545e2895505e29ad60c WatchSource:0}: Error finding container 401c7a168ef49fb893c137be8ecad1c0b17646efeddb5545e2895505e29ad60c: Status 404 returned error can't find the container with id 401c7a168ef49fb893c137be8ecad1c0b17646efeddb5545e2895505e29ad60c
Jul 28 21:14:40.936123 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:40.936099    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw"]
Jul 28 21:14:40.938156 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:14:40.938131    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351892c8_68a8_4218_b95e_d20be76ab83b.slice/crio-008cba8148c2524e08dbe478a63abd35354bb8e297b1618c9c4886639702156d WatchSource:0}: Error finding container 008cba8148c2524e08dbe478a63abd35354bb8e297b1618c9c4886639702156d: Status 404 returned error can't find the container with id 008cba8148c2524e08dbe478a63abd35354bb8e297b1618c9c4886639702156d
Jul 28 21:14:41.500593 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:41.500550    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw" event={"ID":"351892c8-68a8-4218-b95e-d20be76ab83b","Type":"ContainerStarted","Data":"6a55dc58407da8f0eb5ff0946cbbefc3bc0ab438fd8c711f0bcb180d23039c27"}
Jul 28 21:14:41.501059 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:41.500621    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw"
Jul 28 21:14:41.501059 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:41.500670    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw" event={"ID":"351892c8-68a8-4218-b95e-d20be76ab83b","Type":"ContainerStarted","Data":"008cba8148c2524e08dbe478a63abd35354bb8e297b1618c9c4886639702156d"}
Jul 28 21:14:41.502160 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:41.502132    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2" event={"ID":"77100a69-c174-4476-b7d2-32fd0b424d3c","Type":"ContainerStarted","Data":"45a5c2474eaa97f0eeec93127037f04eb23bd263b21feccb5f33079d91e5a9ab"}
Jul 28 21:14:41.502294 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:41.502176    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2" event={"ID":"77100a69-c174-4476-b7d2-32fd0b424d3c","Type":"ContainerStarted","Data":"401c7a168ef49fb893c137be8ecad1c0b17646efeddb5545e2895505e29ad60c"}
Jul 28 21:14:41.502294 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:41.502242    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2"
Jul 28 21:14:41.515606 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:41.515556    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw" podStartSLOduration=1.515540568 podStartE2EDuration="1.515540568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:14:41.513499311 +0000 UTC m=+1636.737842294" watchObservedRunningTime="2026-07-28 21:14:41.515540568 +0000 UTC m=+1636.739883554"
Jul 28 21:14:41.526376 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:41.526338    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2" podStartSLOduration=1.5263206440000001 podStartE2EDuration="1.526320644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:14:41.525593553 +0000 UTC m=+1636.749936535" watchObservedRunningTime="2026-07-28 21:14:41.526320644 +0000 UTC m=+1636.750663632"
Jul 28 21:14:51.157321 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:51.157284    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:14:52.509329 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:52.509298    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw"
Jul 28 21:14:52.509788 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:14:52.509379    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2"
Jul 28 21:15:00.143549 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:00.143519    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-3ba07b96-29754555-2826s"]
Jul 28 21:15:00.146823 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:00.146806    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-3ba07b96-29754555-2826s"
Jul 28 21:15:00.146920 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:00.146856    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754555-2jgkr"]
Jul 28 21:15:00.149093 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:00.149065    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w22xz\""
Jul 28 21:15:00.150742 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:00.150723    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754555-2jgkr"
Jul 28 21:15:00.156269 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:00.155928    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754555-2jgkr"]
Jul 28 21:15:00.161525 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:00.161500    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-3ba07b96-29754555-2826s"]
Jul 28 21:15:00.161657 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:00.161578    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-3ba07b96-29754555-2826s"
Jul 28 21:15:00.170067 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:00.170046    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754555-2jgkr"
Jul 28 21:15:00.306171 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:00.306083    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-3ba07b96-29754555-2826s"]
Jul 28 21:15:00.308817 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:15:00.308794    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d060f2_4c7c_4b66_9028_bc619223c271.slice/crio-d610e567778e626eab82847537953cbeb6afb4e25e270ed928452f7de0c9f761 WatchSource:0}: Error finding container d610e567778e626eab82847537953cbeb6afb4e25e270ed928452f7de0c9f761: Status 404 returned error can't find the container with id d610e567778e626eab82847537953cbeb6afb4e25e270ed928452f7de0c9f761
Jul 28 21:15:00.536709 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:00.536683    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754555-2jgkr"]
Jul 28 21:15:00.541750 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:15:00.541710    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76793ab_4208_4f93_852b_578e523df8b1.slice/crio-58be073add3445abd4c853d53f3df9fff647594d9a0320193ad46467d70edc3b WatchSource:0}: Error finding container 58be073add3445abd4c853d53f3df9fff647594d9a0320193ad46467d70edc3b: Status 404 returned error can't find the container with id 58be073add3445abd4c853d53f3df9fff647594d9a0320193ad46467d70edc3b
Jul 28 21:15:00.602015 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:00.601982    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-3ba07b96-29754555-2826s" event={"ID":"12d060f2-4c7c-4b66-9028-bc619223c271","Type":"ContainerStarted","Data":"d610e567778e626eab82847537953cbeb6afb4e25e270ed928452f7de0c9f761"}
Jul 28 21:15:00.603146 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:00.603118    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754555-2jgkr" event={"ID":"a76793ab-4208-4f93-852b-578e523df8b1","Type":"ContainerStarted","Data":"58be073add3445abd4c853d53f3df9fff647594d9a0320193ad46467d70edc3b"}
Jul 28 21:15:01.609708 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:01.609597    2565 generic.go:356] "Generic (PLEG): container finished" podID="a76793ab-4208-4f93-852b-578e523df8b1" containerID="cb4a547e37e644091e87ac8b592d2bfa5494ff74b0a2288abd2b8bc025bb7601" exitCode=0
Jul 28 21:15:01.609708 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:01.609673    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754555-2jgkr" event={"ID":"a76793ab-4208-4f93-852b-578e523df8b1","Type":"ContainerDied","Data":"cb4a547e37e644091e87ac8b592d2bfa5494ff74b0a2288abd2b8bc025bb7601"}
Jul 28 21:15:01.611332 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:01.611311    2565 generic.go:356] "Generic (PLEG): container finished" podID="12d060f2-4c7c-4b66-9028-bc619223c271" containerID="ee4f1a976cbee7870dd72ae82b69439223f6a51c1e58ecf0a2f17158961ad12f" exitCode=0
Jul 28 21:15:01.611462 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:01.611368    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-3ba07b96-29754555-2826s" event={"ID":"12d060f2-4c7c-4b66-9028-bc619223c271","Type":"ContainerDied","Data":"ee4f1a976cbee7870dd72ae82b69439223f6a51c1e58ecf0a2f17158961ad12f"}
Jul 28 21:15:02.744489 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:02.744464    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754555-2jgkr"
Jul 28 21:15:02.772410 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:02.772382    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-3ba07b96-29754555-2826s"
Jul 28 21:15:03.624205 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:03.624116    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-3ba07b96-29754555-2826s"
Jul 28 21:15:03.624205 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:03.624150    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-3ba07b96-29754555-2826s" event={"ID":"12d060f2-4c7c-4b66-9028-bc619223c271","Type":"ContainerDied","Data":"d610e567778e626eab82847537953cbeb6afb4e25e270ed928452f7de0c9f761"}
Jul 28 21:15:03.624439 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:03.624206    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d610e567778e626eab82847537953cbeb6afb4e25e270ed928452f7de0c9f761"
Jul 28 21:15:03.625482 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:03.625459    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754555-2jgkr" event={"ID":"a76793ab-4208-4f93-852b-578e523df8b1","Type":"ContainerDied","Data":"58be073add3445abd4c853d53f3df9fff647594d9a0320193ad46467d70edc3b"}
Jul 28 21:15:03.625612 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:03.625495    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58be073add3445abd4c853d53f3df9fff647594d9a0320193ad46467d70edc3b"
Jul 28 21:15:03.625612 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:03.625516    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754555-2jgkr"
Jul 28 21:15:04.404093 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:04.404060    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ba07b96-57qtw"]
Jul 28 21:15:04.404464 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:04.404399    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12d060f2-4c7c-4b66-9028-bc619223c271" containerName="cleanup"
Jul 28 21:15:04.404464 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:04.404410    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="12d060f2-4c7c-4b66-9028-bc619223c271" containerName="cleanup"
Jul 28 21:15:04.404464 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:04.404426    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a76793ab-4208-4f93-852b-578e523df8b1" containerName="cleanup"
Jul 28 21:15:04.404464 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:04.404432    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a76793ab-4208-4f93-852b-578e523df8b1" containerName="cleanup"
Jul 28 21:15:04.404605 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:04.404483    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="12d060f2-4c7c-4b66-9028-bc619223c271" containerName="cleanup"
Jul 28 21:15:04.404605 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:04.404493    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a76793ab-4208-4f93-852b-578e523df8b1" containerName="cleanup"
Jul 28 21:15:04.407443 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:04.407428    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ba07b96-57qtw"
Jul 28 21:15:04.409179 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:04.409158    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 21:15:04.409299 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:04.409197    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w22xz\""
Jul 28 21:15:04.415347 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:04.415320    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ba07b96-57qtw"]
Jul 28 21:15:04.465811 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:04.465770    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62470821-68fa-48cd-b36d-867a5879b2af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3ba07b96-57qtw\" (UID: \"62470821-68fa-48cd-b36d-867a5879b2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ba07b96-57qtw"
Jul 28 21:15:04.567126 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:04.567094    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62470821-68fa-48cd-b36d-867a5879b2af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3ba07b96-57qtw\" (UID: \"62470821-68fa-48cd-b36d-867a5879b2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ba07b96-57qtw"
Jul 28 21:15:04.567779 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:04.567758    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62470821-68fa-48cd-b36d-867a5879b2af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3ba07b96-57qtw\" (UID: \"62470821-68fa-48cd-b36d-867a5879b2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ba07b96-57qtw"
Jul 28 21:15:04.719736 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:04.719702    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ba07b96-57qtw"
Jul 28 21:15:04.857422 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:04.857400    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ba07b96-57qtw"]
Jul 28 21:15:04.860061 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:15:04.860035    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62470821_68fa_48cd_b36d_867a5879b2af.slice/crio-0814037d848cf6a1f72686dfe1650886e47ee208a531a9e29ed082a735065af4 WatchSource:0}: Error finding container 0814037d848cf6a1f72686dfe1650886e47ee208a531a9e29ed082a735065af4: Status 404 returned error can't find the container with id 0814037d848cf6a1f72686dfe1650886e47ee208a531a9e29ed082a735065af4
Jul 28 21:15:05.637814 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:05.637773    2565 generic.go:356] "Generic (PLEG): container finished" podID="62470821-68fa-48cd-b36d-867a5879b2af" containerID="e19be05242100e03811f73cacc286e1e78a750ab8f04ac1f27e7fa0df57fda91" exitCode=0
Jul 28 21:15:05.638206 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:05.637861    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ba07b96-57qtw" event={"ID":"62470821-68fa-48cd-b36d-867a5879b2af","Type":"ContainerDied","Data":"e19be05242100e03811f73cacc286e1e78a750ab8f04ac1f27e7fa0df57fda91"}
Jul 28 21:15:05.638206 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:05.637909    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ba07b96-57qtw" event={"ID":"62470821-68fa-48cd-b36d-867a5879b2af","Type":"ContainerStarted","Data":"0814037d848cf6a1f72686dfe1650886e47ee208a531a9e29ed082a735065af4"}
Jul 28 21:15:06.774318 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:06.774297    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ba07b96-57qtw"
Jul 28 21:15:06.887876 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:06.887841    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62470821-68fa-48cd-b36d-867a5879b2af-maas-api-service-ca\") pod \"62470821-68fa-48cd-b36d-867a5879b2af\" (UID: \"62470821-68fa-48cd-b36d-867a5879b2af\") "
Jul 28 21:15:06.888213 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:06.888189    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62470821-68fa-48cd-b36d-867a5879b2af-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62470821-68fa-48cd-b36d-867a5879b2af" (UID: "62470821-68fa-48cd-b36d-867a5879b2af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:15:06.989394 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:06.989362    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62470821-68fa-48cd-b36d-867a5879b2af-maas-api-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:15:07.650144 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:07.650059    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ba07b96-57qtw"
Jul 28 21:15:07.650144 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:07.650098    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ba07b96-57qtw" event={"ID":"62470821-68fa-48cd-b36d-867a5879b2af","Type":"ContainerDied","Data":"0814037d848cf6a1f72686dfe1650886e47ee208a531a9e29ed082a735065af4"}
Jul 28 21:15:07.650144 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:07.650143    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0814037d848cf6a1f72686dfe1650886e47ee208a531a9e29ed082a735065af4"
Jul 28 21:15:10.893551 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:10.893517    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-3ba07b96-29754555-2826s"]
Jul 28 21:15:10.897935 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:10.897901    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-3ba07b96-29754555-2826s"]
Jul 28 21:15:10.924672 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:10.924622    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw"]
Jul 28 21:15:10.924975 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:10.924915    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw" podUID="351892c8-68a8-4218-b95e-d20be76ab83b" containerName="payload-processing" containerID="cri-o://6a55dc58407da8f0eb5ff0946cbbefc3bc0ab438fd8c711f0bcb180d23039c27" gracePeriod=30
Jul 28 21:15:10.931396 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:10.931350    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2"]
Jul 28 21:15:10.931593 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:10.931562    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2" podUID="77100a69-c174-4476-b7d2-32fd0b424d3c" containerName="payload-pre-processing" containerID="cri-o://45a5c2474eaa97f0eeec93127037f04eb23bd263b21feccb5f33079d91e5a9ab" gracePeriod=30
Jul 28 21:15:11.059278 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.059241    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ba07b96-57qtw"]
Jul 28 21:15:11.061340 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.061309    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ba07b96-57qtw"]
Jul 28 21:15:11.209350 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.209327    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw"
Jul 28 21:15:11.212307 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.212282    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2"
Jul 28 21:15:11.325159 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.325129    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/351892c8-68a8-4218-b95e-d20be76ab83b-plugins-config-volume\") pod \"351892c8-68a8-4218-b95e-d20be76ab83b\" (UID: \"351892c8-68a8-4218-b95e-d20be76ab83b\") "
Jul 28 21:15:11.325336 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.325173    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77100a69-c174-4476-b7d2-32fd0b424d3c-plugins-config-volume\") pod \"77100a69-c174-4476-b7d2-32fd0b424d3c\" (UID: \"77100a69-c174-4476-b7d2-32fd0b424d3c\") "
Jul 28 21:15:11.325336 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.325216    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdrjs\" (UniqueName: \"kubernetes.io/projected/351892c8-68a8-4218-b95e-d20be76ab83b-kube-api-access-zdrjs\") pod \"351892c8-68a8-4218-b95e-d20be76ab83b\" (UID: \"351892c8-68a8-4218-b95e-d20be76ab83b\") "
Jul 28 21:15:11.325336 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.325272    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr7z7\" (UniqueName: \"kubernetes.io/projected/77100a69-c174-4476-b7d2-32fd0b424d3c-kube-api-access-wr7z7\") pod \"77100a69-c174-4476-b7d2-32fd0b424d3c\" (UID: \"77100a69-c174-4476-b7d2-32fd0b424d3c\") "
Jul 28 21:15:11.325742 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.325558    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351892c8-68a8-4218-b95e-d20be76ab83b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "351892c8-68a8-4218-b95e-d20be76ab83b" (UID: "351892c8-68a8-4218-b95e-d20be76ab83b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:15:11.325742 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.325605    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77100a69-c174-4476-b7d2-32fd0b424d3c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "77100a69-c174-4476-b7d2-32fd0b424d3c" (UID: "77100a69-c174-4476-b7d2-32fd0b424d3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:15:11.327401 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.327378    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351892c8-68a8-4218-b95e-d20be76ab83b-kube-api-access-zdrjs" (OuterVolumeSpecName: "kube-api-access-zdrjs") pod "351892c8-68a8-4218-b95e-d20be76ab83b" (UID: "351892c8-68a8-4218-b95e-d20be76ab83b"). InnerVolumeSpecName "kube-api-access-zdrjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:15:11.327523 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.327397    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77100a69-c174-4476-b7d2-32fd0b424d3c-kube-api-access-wr7z7" (OuterVolumeSpecName: "kube-api-access-wr7z7") pod "77100a69-c174-4476-b7d2-32fd0b424d3c" (UID: "77100a69-c174-4476-b7d2-32fd0b424d3c"). InnerVolumeSpecName "kube-api-access-wr7z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:15:11.331978 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.331953    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12d060f2-4c7c-4b66-9028-bc619223c271" path="/var/lib/kubelet/pods/12d060f2-4c7c-4b66-9028-bc619223c271/volumes"
Jul 28 21:15:11.332288 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.332270    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62470821-68fa-48cd-b36d-867a5879b2af" path="/var/lib/kubelet/pods/62470821-68fa-48cd-b36d-867a5879b2af/volumes"
Jul 28 21:15:11.426297 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.426259    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/351892c8-68a8-4218-b95e-d20be76ab83b-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:15:11.426297 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.426293    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77100a69-c174-4476-b7d2-32fd0b424d3c-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:15:11.426501 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.426307    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdrjs\" (UniqueName: \"kubernetes.io/projected/351892c8-68a8-4218-b95e-d20be76ab83b-kube-api-access-zdrjs\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:15:11.426501 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.426321    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr7z7\" (UniqueName: \"kubernetes.io/projected/77100a69-c174-4476-b7d2-32fd0b424d3c-kube-api-access-wr7z7\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:15:11.673504 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.673411    2565 generic.go:356] "Generic (PLEG): container finished" podID="351892c8-68a8-4218-b95e-d20be76ab83b" containerID="6a55dc58407da8f0eb5ff0946cbbefc3bc0ab438fd8c711f0bcb180d23039c27" exitCode=0
Jul 28 21:15:11.673504 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.673458    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw"
Jul 28 21:15:11.673748 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.673495    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw" event={"ID":"351892c8-68a8-4218-b95e-d20be76ab83b","Type":"ContainerDied","Data":"6a55dc58407da8f0eb5ff0946cbbefc3bc0ab438fd8c711f0bcb180d23039c27"}
Jul 28 21:15:11.673748 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.673547    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw" event={"ID":"351892c8-68a8-4218-b95e-d20be76ab83b","Type":"ContainerDied","Data":"008cba8148c2524e08dbe478a63abd35354bb8e297b1618c9c4886639702156d"}
Jul 28 21:15:11.673748 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.673566    2565 scope.go:117] "RemoveContainer" containerID="6a55dc58407da8f0eb5ff0946cbbefc3bc0ab438fd8c711f0bcb180d23039c27"
Jul 28 21:15:11.674928 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.674907    2565 generic.go:356] "Generic (PLEG): container finished" podID="77100a69-c174-4476-b7d2-32fd0b424d3c" containerID="45a5c2474eaa97f0eeec93127037f04eb23bd263b21feccb5f33079d91e5a9ab" exitCode=0
Jul 28 21:15:11.675051 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.674937    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2" event={"ID":"77100a69-c174-4476-b7d2-32fd0b424d3c","Type":"ContainerDied","Data":"45a5c2474eaa97f0eeec93127037f04eb23bd263b21feccb5f33079d91e5a9ab"}
Jul 28 21:15:11.675051 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.674951    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2"
Jul 28 21:15:11.675051 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.674974    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2" event={"ID":"77100a69-c174-4476-b7d2-32fd0b424d3c","Type":"ContainerDied","Data":"401c7a168ef49fb893c137be8ecad1c0b17646efeddb5545e2895505e29ad60c"}
Jul 28 21:15:11.682698 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.682679    2565 scope.go:117] "RemoveContainer" containerID="6a55dc58407da8f0eb5ff0946cbbefc3bc0ab438fd8c711f0bcb180d23039c27"
Jul 28 21:15:11.682965 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:15:11.682942    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a55dc58407da8f0eb5ff0946cbbefc3bc0ab438fd8c711f0bcb180d23039c27\": container with ID starting with 6a55dc58407da8f0eb5ff0946cbbefc3bc0ab438fd8c711f0bcb180d23039c27 not found: ID does not exist" containerID="6a55dc58407da8f0eb5ff0946cbbefc3bc0ab438fd8c711f0bcb180d23039c27"
Jul 28 21:15:11.683048 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.682977    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a55dc58407da8f0eb5ff0946cbbefc3bc0ab438fd8c711f0bcb180d23039c27"} err="failed to get container status \"6a55dc58407da8f0eb5ff0946cbbefc3bc0ab438fd8c711f0bcb180d23039c27\": rpc error: code = NotFound desc = could not find container \"6a55dc58407da8f0eb5ff0946cbbefc3bc0ab438fd8c711f0bcb180d23039c27\": container with ID starting with 6a55dc58407da8f0eb5ff0946cbbefc3bc0ab438fd8c711f0bcb180d23039c27 not found: ID does not exist"
Jul 28 21:15:11.683048 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.683000    2565 scope.go:117] "RemoveContainer" containerID="45a5c2474eaa97f0eeec93127037f04eb23bd263b21feccb5f33079d91e5a9ab"
Jul 28 21:15:11.691096 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.691073    2565 scope.go:117] "RemoveContainer" containerID="45a5c2474eaa97f0eeec93127037f04eb23bd263b21feccb5f33079d91e5a9ab"
Jul 28 21:15:11.691378 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:15:11.691354    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a5c2474eaa97f0eeec93127037f04eb23bd263b21feccb5f33079d91e5a9ab\": container with ID starting with 45a5c2474eaa97f0eeec93127037f04eb23bd263b21feccb5f33079d91e5a9ab not found: ID does not exist" containerID="45a5c2474eaa97f0eeec93127037f04eb23bd263b21feccb5f33079d91e5a9ab"
Jul 28 21:15:11.691437 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.691388    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a5c2474eaa97f0eeec93127037f04eb23bd263b21feccb5f33079d91e5a9ab"} err="failed to get container status \"45a5c2474eaa97f0eeec93127037f04eb23bd263b21feccb5f33079d91e5a9ab\": rpc error: code = NotFound desc = could not find container \"45a5c2474eaa97f0eeec93127037f04eb23bd263b21feccb5f33079d91e5a9ab\": container with ID starting with 45a5c2474eaa97f0eeec93127037f04eb23bd263b21feccb5f33079d91e5a9ab not found: ID does not exist"
Jul 28 21:15:11.691518 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.691504    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw"]
Jul 28 21:15:11.695397 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.695371    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3ba07b96-54898d8b45-6p4rw"]
Jul 28 21:15:11.704497 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.704476    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2"]
Jul 28 21:15:11.707506 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:11.707485    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3ba07b96-959cf64fb-p7lc2"]
Jul 28 21:15:13.331731 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:13.331701    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="351892c8-68a8-4218-b95e-d20be76ab83b" path="/var/lib/kubelet/pods/351892c8-68a8-4218-b95e-d20be76ab83b/volumes"
Jul 28 21:15:13.332104 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:13.332080    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77100a69-c174-4476-b7d2-32fd0b424d3c" path="/var/lib/kubelet/pods/77100a69-c174-4476-b7d2-32fd0b424d3c/volumes"
Jul 28 21:15:14.613232 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:14.613200    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:15:15.187676 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.185948    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf"]
Jul 28 21:15:15.187676 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.186631    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77100a69-c174-4476-b7d2-32fd0b424d3c" containerName="payload-pre-processing"
Jul 28 21:15:15.187676 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.186668    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="77100a69-c174-4476-b7d2-32fd0b424d3c" containerName="payload-pre-processing"
Jul 28 21:15:15.187676 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.186690    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="351892c8-68a8-4218-b95e-d20be76ab83b" containerName="payload-processing"
Jul 28 21:15:15.187676 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.186698    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="351892c8-68a8-4218-b95e-d20be76ab83b" containerName="payload-processing"
Jul 28 21:15:15.187676 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.186735    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62470821-68fa-48cd-b36d-867a5879b2af" containerName="revoke-keys"
Jul 28 21:15:15.187676 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.186743    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62470821-68fa-48cd-b36d-867a5879b2af" containerName="revoke-keys"
Jul 28 21:15:15.187676 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.186898    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="351892c8-68a8-4218-b95e-d20be76ab83b" containerName="payload-processing"
Jul 28 21:15:15.187676 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.186918    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62470821-68fa-48cd-b36d-867a5879b2af" containerName="revoke-keys"
Jul 28 21:15:15.187676 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.186929    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="77100a69-c174-4476-b7d2-32fd0b424d3c" containerName="payload-pre-processing"
Jul 28 21:15:15.192269 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.192243    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf"
Jul 28 21:15:15.194748 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.194721    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-37fe5a92\""
Jul 28 21:15:15.194966 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.194943    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-37fe5a92-dockercfg-7vrms\""
Jul 28 21:15:15.197335 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.197313    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf"]
Jul 28 21:15:15.203433 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.203407    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq"]
Jul 28 21:15:15.207016 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.206999    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq"
Jul 28 21:15:15.218038 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.218015    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq"]
Jul 28 21:15:15.260470 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.260435    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snswd\" (UniqueName: \"kubernetes.io/projected/70c47f14-efc2-404c-b725-0b312e9786b2-kube-api-access-snswd\") pod \"payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq\" (UID: \"70c47f14-efc2-404c-b725-0b312e9786b2\") " pod="openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq"
Jul 28 21:15:15.260470 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.260471    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll6vb\" (UniqueName: \"kubernetes.io/projected/5462ed20-8269-4144-bb95-7193bc16f3bb-kube-api-access-ll6vb\") pod \"payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf\" (UID: \"5462ed20-8269-4144-bb95-7193bc16f3bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf"
Jul 28 21:15:15.260713 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.260577    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5462ed20-8269-4144-bb95-7193bc16f3bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf\" (UID: \"5462ed20-8269-4144-bb95-7193bc16f3bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf"
Jul 28 21:15:15.260713 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.260654    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70c47f14-efc2-404c-b725-0b312e9786b2-plugins-config-volume\") pod \"payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq\" (UID: \"70c47f14-efc2-404c-b725-0b312e9786b2\") " pod="openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq"
Jul 28 21:15:15.361504 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.361469    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70c47f14-efc2-404c-b725-0b312e9786b2-plugins-config-volume\") pod \"payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq\" (UID: \"70c47f14-efc2-404c-b725-0b312e9786b2\") " pod="openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq"
Jul 28 21:15:15.361504 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.361508    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snswd\" (UniqueName: \"kubernetes.io/projected/70c47f14-efc2-404c-b725-0b312e9786b2-kube-api-access-snswd\") pod \"payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq\" (UID: \"70c47f14-efc2-404c-b725-0b312e9786b2\") " pod="openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq"
Jul 28 21:15:15.361740 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.361526    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll6vb\" (UniqueName: \"kubernetes.io/projected/5462ed20-8269-4144-bb95-7193bc16f3bb-kube-api-access-ll6vb\") pod \"payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf\" (UID: \"5462ed20-8269-4144-bb95-7193bc16f3bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf"
Jul 28 21:15:15.361740 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.361578    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5462ed20-8269-4144-bb95-7193bc16f3bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf\" (UID: \"5462ed20-8269-4144-bb95-7193bc16f3bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf"
Jul 28 21:15:15.362247 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.362228    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5462ed20-8269-4144-bb95-7193bc16f3bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf\" (UID: \"5462ed20-8269-4144-bb95-7193bc16f3bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf"
Jul 28 21:15:15.362285 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.362242    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70c47f14-efc2-404c-b725-0b312e9786b2-plugins-config-volume\") pod \"payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq\" (UID: \"70c47f14-efc2-404c-b725-0b312e9786b2\") " pod="openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq"
Jul 28 21:15:15.368876 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.368846    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snswd\" (UniqueName: \"kubernetes.io/projected/70c47f14-efc2-404c-b725-0b312e9786b2-kube-api-access-snswd\") pod \"payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq\" (UID: \"70c47f14-efc2-404c-b725-0b312e9786b2\") " pod="openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq"
Jul 28 21:15:15.368985 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.368877    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll6vb\" (UniqueName: \"kubernetes.io/projected/5462ed20-8269-4144-bb95-7193bc16f3bb-kube-api-access-ll6vb\") pod \"payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf\" (UID: \"5462ed20-8269-4144-bb95-7193bc16f3bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf"
Jul 28 21:15:15.505303 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.505204    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf"
Jul 28 21:15:15.517568 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.517538    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq"
Jul 28 21:15:15.647862 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.647831    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf"]
Jul 28 21:15:15.650095 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:15:15.650069    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5462ed20_8269_4144_bb95_7193bc16f3bb.slice/crio-cd137c9208e9801947a2619fc000ae5774283be1c092071e2c6d1b5d751f9b5a WatchSource:0}: Error finding container cd137c9208e9801947a2619fc000ae5774283be1c092071e2c6d1b5d751f9b5a: Status 404 returned error can't find the container with id cd137c9208e9801947a2619fc000ae5774283be1c092071e2c6d1b5d751f9b5a
Jul 28 21:15:15.675768 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.675742    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq"]
Jul 28 21:15:15.682792 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:15:15.682761    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c47f14_efc2_404c_b725_0b312e9786b2.slice/crio-db5ed7df31b0c77814730116669909e790d09e3683f8c54fbec791f7b7cb8715 WatchSource:0}: Error finding container db5ed7df31b0c77814730116669909e790d09e3683f8c54fbec791f7b7cb8715: Status 404 returned error can't find the container with id db5ed7df31b0c77814730116669909e790d09e3683f8c54fbec791f7b7cb8715
Jul 28 21:15:15.700157 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.700132    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq" event={"ID":"70c47f14-efc2-404c-b725-0b312e9786b2","Type":"ContainerStarted","Data":"db5ed7df31b0c77814730116669909e790d09e3683f8c54fbec791f7b7cb8715"}
Jul 28 21:15:15.701493 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:15.701465    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf" event={"ID":"5462ed20-8269-4144-bb95-7193bc16f3bb","Type":"ContainerStarted","Data":"cd137c9208e9801947a2619fc000ae5774283be1c092071e2c6d1b5d751f9b5a"}
Jul 28 21:15:16.708576 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:16.708540    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq" event={"ID":"70c47f14-efc2-404c-b725-0b312e9786b2","Type":"ContainerStarted","Data":"764907f515eb0700daf341ee569ccc4e2befa41e476cf380011a782d29dd362d"}
Jul 28 21:15:16.709037 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:16.708728    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq"
Jul 28 21:15:16.710117 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:16.710097    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf" event={"ID":"5462ed20-8269-4144-bb95-7193bc16f3bb","Type":"ContainerStarted","Data":"a238edc4ca06682945679f9d893f6f2c6427b3906ad7568c38054d76a6146075"}
Jul 28 21:15:16.710228 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:16.710217    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf"
Jul 28 21:15:16.728424 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:16.728384    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq" podStartSLOduration=1.728367361 podStartE2EDuration="1.728367361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:15:16.726047154 +0000 UTC m=+1671.950390142" watchObservedRunningTime="2026-07-28 21:15:16.728367361 +0000 UTC m=+1671.952710325"
Jul 28 21:15:16.741216 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:16.741177    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf" podStartSLOduration=1.741164331 podStartE2EDuration="1.741164331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:15:16.739560324 +0000 UTC m=+1671.963903310" watchObservedRunningTime="2026-07-28 21:15:16.741164331 +0000 UTC m=+1671.965507369"
Jul 28 21:15:24.671323 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:24.671290    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:15:27.717208 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:27.717174    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq"
Jul 28 21:15:27.717604 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:27.717251    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf"
Jul 28 21:15:40.039956 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:40.039921    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37fe5a92-sd4k2"]
Jul 28 21:15:40.047014 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:40.045996    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37fe5a92-sd4k2"
Jul 28 21:15:40.049730 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:40.049707    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 21:15:40.049896 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:40.049747    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w22xz\""
Jul 28 21:15:40.051007 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:40.050983    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37fe5a92-sd4k2"]
Jul 28 21:15:40.178494 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:40.178442    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc832b1-2c73-4fb5-8fde-07872873a38a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-37fe5a92-sd4k2\" (UID: \"2bc832b1-2c73-4fb5-8fde-07872873a38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37fe5a92-sd4k2"
Jul 28 21:15:40.278964 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:40.278932    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc832b1-2c73-4fb5-8fde-07872873a38a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-37fe5a92-sd4k2\" (UID: \"2bc832b1-2c73-4fb5-8fde-07872873a38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37fe5a92-sd4k2"
Jul 28 21:15:40.279600 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:40.279578    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc832b1-2c73-4fb5-8fde-07872873a38a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-37fe5a92-sd4k2\" (UID: \"2bc832b1-2c73-4fb5-8fde-07872873a38a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37fe5a92-sd4k2"
Jul 28 21:15:40.358963 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:40.358887    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37fe5a92-sd4k2"
Jul 28 21:15:40.494938 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:40.494911    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37fe5a92-sd4k2"]
Jul 28 21:15:40.496759 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:15:40.496727    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc832b1_2c73_4fb5_8fde_07872873a38a.slice/crio-426f953456d7ab6fb3c71dd0ea36a323f8c9d4ffca7d2a5bb2864fe142ad5eed WatchSource:0}: Error finding container 426f953456d7ab6fb3c71dd0ea36a323f8c9d4ffca7d2a5bb2864fe142ad5eed: Status 404 returned error can't find the container with id 426f953456d7ab6fb3c71dd0ea36a323f8c9d4ffca7d2a5bb2864fe142ad5eed
Jul 28 21:15:40.843954 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:40.843924    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37fe5a92-sd4k2" event={"ID":"2bc832b1-2c73-4fb5-8fde-07872873a38a","Type":"ContainerStarted","Data":"426f953456d7ab6fb3c71dd0ea36a323f8c9d4ffca7d2a5bb2864fe142ad5eed"}
Jul 28 21:15:41.850425 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:41.850385    2565 generic.go:356] "Generic (PLEG): container finished" podID="2bc832b1-2c73-4fb5-8fde-07872873a38a" containerID="630a361bff191fd764bbc87cfcccbc4a4cb19130f871fb9fe7b69bf2a12dec2c" exitCode=0
Jul 28 21:15:41.850918 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:41.850450    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37fe5a92-sd4k2" event={"ID":"2bc832b1-2c73-4fb5-8fde-07872873a38a","Type":"ContainerDied","Data":"630a361bff191fd764bbc87cfcccbc4a4cb19130f871fb9fe7b69bf2a12dec2c"}
Jul 28 21:15:42.987973 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:42.987947    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37fe5a92-sd4k2"
Jul 28 21:15:43.102212 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:43.102175    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc832b1-2c73-4fb5-8fde-07872873a38a-maas-api-service-ca\") pod \"2bc832b1-2c73-4fb5-8fde-07872873a38a\" (UID: \"2bc832b1-2c73-4fb5-8fde-07872873a38a\") "
Jul 28 21:15:43.102520 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:43.102496    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc832b1-2c73-4fb5-8fde-07872873a38a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2bc832b1-2c73-4fb5-8fde-07872873a38a" (UID: "2bc832b1-2c73-4fb5-8fde-07872873a38a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:15:43.203549 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:43.203518    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bc832b1-2c73-4fb5-8fde-07872873a38a-maas-api-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:15:43.862515 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:43.862479    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37fe5a92-sd4k2" event={"ID":"2bc832b1-2c73-4fb5-8fde-07872873a38a","Type":"ContainerDied","Data":"426f953456d7ab6fb3c71dd0ea36a323f8c9d4ffca7d2a5bb2864fe142ad5eed"}
Jul 28 21:15:43.862515 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:43.862512    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37fe5a92-sd4k2"
Jul 28 21:15:43.862763 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:43.862528    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="426f953456d7ab6fb3c71dd0ea36a323f8c9d4ffca7d2a5bb2864fe142ad5eed"
Jul 28 21:15:45.219652 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.219606    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq"]
Jul 28 21:15:45.220140 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.219862    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq" podUID="70c47f14-efc2-404c-b725-0b312e9786b2" containerName="payload-processing" containerID="cri-o://764907f515eb0700daf341ee569ccc4e2befa41e476cf380011a782d29dd362d" gracePeriod=30
Jul 28 21:15:45.235189 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.235153    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf"]
Jul 28 21:15:45.235406 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.235377    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf" podUID="5462ed20-8269-4144-bb95-7193bc16f3bb" containerName="payload-pre-processing" containerID="cri-o://a238edc4ca06682945679f9d893f6f2c6427b3906ad7568c38054d76a6146075" gracePeriod=30
Jul 28 21:15:45.379710 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.379674    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37fe5a92-sd4k2"]
Jul 28 21:15:45.383452 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.383422    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37fe5a92-sd4k2"]
Jul 28 21:15:45.488623 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.488598    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq"
Jul 28 21:15:45.542905 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.542874    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf"
Jul 28 21:15:45.625754 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.625719    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70c47f14-efc2-404c-b725-0b312e9786b2-plugins-config-volume\") pod \"70c47f14-efc2-404c-b725-0b312e9786b2\" (UID: \"70c47f14-efc2-404c-b725-0b312e9786b2\") "
Jul 28 21:15:45.625937 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.625792    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snswd\" (UniqueName: \"kubernetes.io/projected/70c47f14-efc2-404c-b725-0b312e9786b2-kube-api-access-snswd\") pod \"70c47f14-efc2-404c-b725-0b312e9786b2\" (UID: \"70c47f14-efc2-404c-b725-0b312e9786b2\") "
Jul 28 21:15:45.626125 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.626102    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c47f14-efc2-404c-b725-0b312e9786b2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70c47f14-efc2-404c-b725-0b312e9786b2" (UID: "70c47f14-efc2-404c-b725-0b312e9786b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:15:45.627931 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.627905    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c47f14-efc2-404c-b725-0b312e9786b2-kube-api-access-snswd" (OuterVolumeSpecName: "kube-api-access-snswd") pod "70c47f14-efc2-404c-b725-0b312e9786b2" (UID: "70c47f14-efc2-404c-b725-0b312e9786b2"). InnerVolumeSpecName "kube-api-access-snswd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:15:45.726469 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.726380    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5462ed20-8269-4144-bb95-7193bc16f3bb-plugins-config-volume\") pod \"5462ed20-8269-4144-bb95-7193bc16f3bb\" (UID: \"5462ed20-8269-4144-bb95-7193bc16f3bb\") "
Jul 28 21:15:45.726469 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.726420    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ll6vb\" (UniqueName: \"kubernetes.io/projected/5462ed20-8269-4144-bb95-7193bc16f3bb-kube-api-access-ll6vb\") pod \"5462ed20-8269-4144-bb95-7193bc16f3bb\" (UID: \"5462ed20-8269-4144-bb95-7193bc16f3bb\") "
Jul 28 21:15:45.726682 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.726594    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snswd\" (UniqueName: \"kubernetes.io/projected/70c47f14-efc2-404c-b725-0b312e9786b2-kube-api-access-snswd\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:15:45.726682 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.726605    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70c47f14-efc2-404c-b725-0b312e9786b2-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:15:45.726825 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.726800    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5462ed20-8269-4144-bb95-7193bc16f3bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5462ed20-8269-4144-bb95-7193bc16f3bb" (UID: "5462ed20-8269-4144-bb95-7193bc16f3bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:15:45.728487 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.728463    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5462ed20-8269-4144-bb95-7193bc16f3bb-kube-api-access-ll6vb" (OuterVolumeSpecName: "kube-api-access-ll6vb") pod "5462ed20-8269-4144-bb95-7193bc16f3bb" (UID: "5462ed20-8269-4144-bb95-7193bc16f3bb"). InnerVolumeSpecName "kube-api-access-ll6vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:15:45.827049 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.827012    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5462ed20-8269-4144-bb95-7193bc16f3bb-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:15:45.827049 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.827042    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll6vb\" (UniqueName: \"kubernetes.io/projected/5462ed20-8269-4144-bb95-7193bc16f3bb-kube-api-access-ll6vb\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:15:45.874469 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.874436    2565 generic.go:356] "Generic (PLEG): container finished" podID="70c47f14-efc2-404c-b725-0b312e9786b2" containerID="764907f515eb0700daf341ee569ccc4e2befa41e476cf380011a782d29dd362d" exitCode=0
Jul 28 21:15:45.874665 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.874483    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq"
Jul 28 21:15:45.874665 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.874517    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq" event={"ID":"70c47f14-efc2-404c-b725-0b312e9786b2","Type":"ContainerDied","Data":"764907f515eb0700daf341ee569ccc4e2befa41e476cf380011a782d29dd362d"}
Jul 28 21:15:45.874665 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.874584    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq" event={"ID":"70c47f14-efc2-404c-b725-0b312e9786b2","Type":"ContainerDied","Data":"db5ed7df31b0c77814730116669909e790d09e3683f8c54fbec791f7b7cb8715"}
Jul 28 21:15:45.874665 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.874614    2565 scope.go:117] "RemoveContainer" containerID="764907f515eb0700daf341ee569ccc4e2befa41e476cf380011a782d29dd362d"
Jul 28 21:15:45.875959 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.875936    2565 generic.go:356] "Generic (PLEG): container finished" podID="5462ed20-8269-4144-bb95-7193bc16f3bb" containerID="a238edc4ca06682945679f9d893f6f2c6427b3906ad7568c38054d76a6146075" exitCode=0
Jul 28 21:15:45.876045 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.875976    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf"
Jul 28 21:15:45.876045 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.875992    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf" event={"ID":"5462ed20-8269-4144-bb95-7193bc16f3bb","Type":"ContainerDied","Data":"a238edc4ca06682945679f9d893f6f2c6427b3906ad7568c38054d76a6146075"}
Jul 28 21:15:45.876045 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.876031    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf" event={"ID":"5462ed20-8269-4144-bb95-7193bc16f3bb","Type":"ContainerDied","Data":"cd137c9208e9801947a2619fc000ae5774283be1c092071e2c6d1b5d751f9b5a"}
Jul 28 21:15:45.884734 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.884710    2565 scope.go:117] "RemoveContainer" containerID="764907f515eb0700daf341ee569ccc4e2befa41e476cf380011a782d29dd362d"
Jul 28 21:15:45.885055 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:15:45.885031    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764907f515eb0700daf341ee569ccc4e2befa41e476cf380011a782d29dd362d\": container with ID starting with 764907f515eb0700daf341ee569ccc4e2befa41e476cf380011a782d29dd362d not found: ID does not exist" containerID="764907f515eb0700daf341ee569ccc4e2befa41e476cf380011a782d29dd362d"
Jul 28 21:15:45.885102 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.885071    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764907f515eb0700daf341ee569ccc4e2befa41e476cf380011a782d29dd362d"} err="failed to get container status \"764907f515eb0700daf341ee569ccc4e2befa41e476cf380011a782d29dd362d\": rpc error: code = NotFound desc = could not find container \"764907f515eb0700daf341ee569ccc4e2befa41e476cf380011a782d29dd362d\": container with ID starting with 764907f515eb0700daf341ee569ccc4e2befa41e476cf380011a782d29dd362d not found: ID does not exist"
Jul 28 21:15:45.885102 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.885090    2565 scope.go:117] "RemoveContainer" containerID="a238edc4ca06682945679f9d893f6f2c6427b3906ad7568c38054d76a6146075"
Jul 28 21:15:45.893099 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.893079    2565 scope.go:117] "RemoveContainer" containerID="a238edc4ca06682945679f9d893f6f2c6427b3906ad7568c38054d76a6146075"
Jul 28 21:15:45.893323 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:15:45.893305    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a238edc4ca06682945679f9d893f6f2c6427b3906ad7568c38054d76a6146075\": container with ID starting with a238edc4ca06682945679f9d893f6f2c6427b3906ad7568c38054d76a6146075 not found: ID does not exist" containerID="a238edc4ca06682945679f9d893f6f2c6427b3906ad7568c38054d76a6146075"
Jul 28 21:15:45.893371 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.893331    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a238edc4ca06682945679f9d893f6f2c6427b3906ad7568c38054d76a6146075"} err="failed to get container status \"a238edc4ca06682945679f9d893f6f2c6427b3906ad7568c38054d76a6146075\": rpc error: code = NotFound desc = could not find container \"a238edc4ca06682945679f9d893f6f2c6427b3906ad7568c38054d76a6146075\": container with ID starting with a238edc4ca06682945679f9d893f6f2c6427b3906ad7568c38054d76a6146075 not found: ID does not exist"
Jul 28 21:15:45.899085 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.899064    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq"]
Jul 28 21:15:45.901127 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.901105    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-37fe5a92-8648548c56-9gfcq"]
Jul 28 21:15:45.910985 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.910962    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf"]
Jul 28 21:15:45.915342 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:45.915321    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-37fe5a92-69f8df86fb-2gbtf"]
Jul 28 21:15:47.332037 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:47.331964    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bc832b1-2c73-4fb5-8fde-07872873a38a" path="/var/lib/kubelet/pods/2bc832b1-2c73-4fb5-8fde-07872873a38a/volumes"
Jul 28 21:15:47.332398 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:47.332294    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5462ed20-8269-4144-bb95-7193bc16f3bb" path="/var/lib/kubelet/pods/5462ed20-8269-4144-bb95-7193bc16f3bb/volumes"
Jul 28 21:15:47.332675 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:47.332659    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70c47f14-efc2-404c-b725-0b312e9786b2" path="/var/lib/kubelet/pods/70c47f14-efc2-404c-b725-0b312e9786b2/volumes"
Jul 28 21:15:49.053082 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:49.053043    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:15:50.605103 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.605069    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4"]
Jul 28 21:15:50.605490 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.605397    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70c47f14-efc2-404c-b725-0b312e9786b2" containerName="payload-processing"
Jul 28 21:15:50.605490 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.605408    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="70c47f14-efc2-404c-b725-0b312e9786b2" containerName="payload-processing"
Jul 28 21:15:50.605490 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.605422    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5462ed20-8269-4144-bb95-7193bc16f3bb" containerName="payload-pre-processing"
Jul 28 21:15:50.605490 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.605428    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5462ed20-8269-4144-bb95-7193bc16f3bb" containerName="payload-pre-processing"
Jul 28 21:15:50.605490 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.605444    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bc832b1-2c73-4fb5-8fde-07872873a38a" containerName="revoke-keys"
Jul 28 21:15:50.605490 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.605451    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bc832b1-2c73-4fb5-8fde-07872873a38a" containerName="revoke-keys"
Jul 28 21:15:50.605695 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.605502    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bc832b1-2c73-4fb5-8fde-07872873a38a" containerName="revoke-keys"
Jul 28 21:15:50.605695 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.605511    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="70c47f14-efc2-404c-b725-0b312e9786b2" containerName="payload-processing"
Jul 28 21:15:50.605695 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.605518    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5462ed20-8269-4144-bb95-7193bc16f3bb" containerName="payload-pre-processing"
Jul 28 21:15:50.609836 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.609810    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4"
Jul 28 21:15:50.611608 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.611579    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-99e39a7a\""
Jul 28 21:15:50.611755 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.611586    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-99e39a7a-dockercfg-zwkjp\""
Jul 28 21:15:50.618494 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.618472    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4"]
Jul 28 21:15:50.621789 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.621770    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs"]
Jul 28 21:15:50.625032 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.625015    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs"
Jul 28 21:15:50.633676 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.633630    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs"]
Jul 28 21:15:50.667585 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.667532    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6994\" (UniqueName: \"kubernetes.io/projected/ee977f22-a286-4d3b-bd86-4a12fa64908d-kube-api-access-l6994\") pod \"payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs\" (UID: \"ee977f22-a286-4d3b-bd86-4a12fa64908d\") " pod="openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs"
Jul 28 21:15:50.667805 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.667612    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08d80d48-a073-46a3-808a-51327ae61595-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4\" (UID: \"08d80d48-a073-46a3-808a-51327ae61595\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4"
Jul 28 21:15:50.667805 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.667659    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee977f22-a286-4d3b-bd86-4a12fa64908d-plugins-config-volume\") pod \"payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs\" (UID: \"ee977f22-a286-4d3b-bd86-4a12fa64908d\") " pod="openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs"
Jul 28 21:15:50.667805 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.667688    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlmrm\" (UniqueName: \"kubernetes.io/projected/08d80d48-a073-46a3-808a-51327ae61595-kube-api-access-nlmrm\") pod \"payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4\" (UID: \"08d80d48-a073-46a3-808a-51327ae61595\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4"
Jul 28 21:15:50.768416 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.768384    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08d80d48-a073-46a3-808a-51327ae61595-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4\" (UID: \"08d80d48-a073-46a3-808a-51327ae61595\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4"
Jul 28 21:15:50.768599 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.768422    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee977f22-a286-4d3b-bd86-4a12fa64908d-plugins-config-volume\") pod \"payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs\" (UID: \"ee977f22-a286-4d3b-bd86-4a12fa64908d\") " pod="openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs"
Jul 28 21:15:50.768599 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.768563    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlmrm\" (UniqueName: \"kubernetes.io/projected/08d80d48-a073-46a3-808a-51327ae61595-kube-api-access-nlmrm\") pod \"payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4\" (UID: \"08d80d48-a073-46a3-808a-51327ae61595\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4"
Jul 28 21:15:50.768753 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.768696    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6994\" (UniqueName: \"kubernetes.io/projected/ee977f22-a286-4d3b-bd86-4a12fa64908d-kube-api-access-l6994\") pod \"payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs\" (UID: \"ee977f22-a286-4d3b-bd86-4a12fa64908d\") " pod="openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs"
Jul 28 21:15:50.769370 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.769348    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee977f22-a286-4d3b-bd86-4a12fa64908d-plugins-config-volume\") pod \"payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs\" (UID: \"ee977f22-a286-4d3b-bd86-4a12fa64908d\") " pod="openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs"
Jul 28 21:15:50.769411 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.769348    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08d80d48-a073-46a3-808a-51327ae61595-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4\" (UID: \"08d80d48-a073-46a3-808a-51327ae61595\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4"
Jul 28 21:15:50.777125 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.777096    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlmrm\" (UniqueName: \"kubernetes.io/projected/08d80d48-a073-46a3-808a-51327ae61595-kube-api-access-nlmrm\") pod \"payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4\" (UID: \"08d80d48-a073-46a3-808a-51327ae61595\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4"
Jul 28 21:15:50.777236 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.777215    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6994\" (UniqueName: \"kubernetes.io/projected/ee977f22-a286-4d3b-bd86-4a12fa64908d-kube-api-access-l6994\") pod \"payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs\" (UID: \"ee977f22-a286-4d3b-bd86-4a12fa64908d\") " pod="openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs"
Jul 28 21:15:50.921127 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.921040    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4"
Jul 28 21:15:50.937326 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:50.937281    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs"
Jul 28 21:15:51.270241 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:51.270207    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4"]
Jul 28 21:15:51.272139 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:15:51.272116    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d80d48_a073_46a3_808a_51327ae61595.slice/crio-a07e0c3410afd08ad9bdfe268f9450f67fc149cb26ad1314446a360819f5604a WatchSource:0}: Error finding container a07e0c3410afd08ad9bdfe268f9450f67fc149cb26ad1314446a360819f5604a: Status 404 returned error can't find the container with id a07e0c3410afd08ad9bdfe268f9450f67fc149cb26ad1314446a360819f5604a
Jul 28 21:15:51.299189 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:51.299168    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs"]
Jul 28 21:15:51.302817 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:15:51.302785    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee977f22_a286_4d3b_bd86_4a12fa64908d.slice/crio-c97a81494ff0e530e30bb776cb0fb7cbfa01c0b2ec063abbb43871bc5d516d34 WatchSource:0}: Error finding container c97a81494ff0e530e30bb776cb0fb7cbfa01c0b2ec063abbb43871bc5d516d34: Status 404 returned error can't find the container with id c97a81494ff0e530e30bb776cb0fb7cbfa01c0b2ec063abbb43871bc5d516d34
Jul 28 21:15:51.912086 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:51.912042    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4" event={"ID":"08d80d48-a073-46a3-808a-51327ae61595","Type":"ContainerStarted","Data":"cb44edda4810cfd793f3b4ae277e6dbe119eb8242a67528b8168d84945ae22f1"}
Jul 28 21:15:51.912555 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:51.912150    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4" event={"ID":"08d80d48-a073-46a3-808a-51327ae61595","Type":"ContainerStarted","Data":"a07e0c3410afd08ad9bdfe268f9450f67fc149cb26ad1314446a360819f5604a"}
Jul 28 21:15:51.912555 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:51.912182    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4"
Jul 28 21:15:51.913521 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:51.913495    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs" event={"ID":"ee977f22-a286-4d3b-bd86-4a12fa64908d","Type":"ContainerStarted","Data":"b6add0cbb7c643867717a9b2703831ae38fd389162a12198433af1e235eddd81"}
Jul 28 21:15:51.913660 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:51.913539    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs" event={"ID":"ee977f22-a286-4d3b-bd86-4a12fa64908d","Type":"ContainerStarted","Data":"c97a81494ff0e530e30bb776cb0fb7cbfa01c0b2ec063abbb43871bc5d516d34"}
Jul 28 21:15:51.913660 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:51.913600    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs"
Jul 28 21:15:51.928700 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:51.928652    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4" podStartSLOduration=1.9286237769999999 podStartE2EDuration="1.928623777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:15:51.927032633 +0000 UTC m=+1707.151375624" watchObservedRunningTime="2026-07-28 21:15:51.928623777 +0000 UTC m=+1707.152966755"
Jul 28 21:15:51.944564 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:15:51.944509    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs" podStartSLOduration=1.9444905289999999 podStartE2EDuration="1.944490529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:15:51.941184034 +0000 UTC m=+1707.165527023" watchObservedRunningTime="2026-07-28 21:15:51.944490529 +0000 UTC m=+1707.168833518"
Jul 28 21:16:00.057883 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:00.057841    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:16:02.921819 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:02.921780    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4"
Jul 28 21:16:02.922250 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:02.921942    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs"
Jul 28 21:16:14.596399 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:14.596361    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99e39a7a-c7456"]
Jul 28 21:16:14.599914 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:14.599898    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99e39a7a-c7456"
Jul 28 21:16:14.605347 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:14.605323    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 21:16:14.605601 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:14.605578    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w22xz\""
Jul 28 21:16:14.611506 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:14.611481    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99e39a7a-c7456"]
Jul 28 21:16:14.638942 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:14.638913    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0041b29-3335-44fd-943e-66db9471a953-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-99e39a7a-c7456\" (UID: \"b0041b29-3335-44fd-943e-66db9471a953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99e39a7a-c7456"
Jul 28 21:16:14.740041 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:14.740000    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0041b29-3335-44fd-943e-66db9471a953-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-99e39a7a-c7456\" (UID: \"b0041b29-3335-44fd-943e-66db9471a953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99e39a7a-c7456"
Jul 28 21:16:14.740685 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:14.740665    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0041b29-3335-44fd-943e-66db9471a953-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-99e39a7a-c7456\" (UID: \"b0041b29-3335-44fd-943e-66db9471a953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99e39a7a-c7456"
Jul 28 21:16:14.911135 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:14.911036    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99e39a7a-c7456"
Jul 28 21:16:15.046519 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:15.046495    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99e39a7a-c7456"]
Jul 28 21:16:16.047857 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:16.047826    2565 generic.go:356] "Generic (PLEG): container finished" podID="b0041b29-3335-44fd-943e-66db9471a953" containerID="d3ad450103f070d8af0f82a64e9a9f95c29cc3389625a11b9ea29b926d8ea10f" exitCode=0
Jul 28 21:16:16.048239 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:16.047898    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99e39a7a-c7456" event={"ID":"b0041b29-3335-44fd-943e-66db9471a953","Type":"ContainerDied","Data":"d3ad450103f070d8af0f82a64e9a9f95c29cc3389625a11b9ea29b926d8ea10f"}
Jul 28 21:16:16.048239 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:16.047936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99e39a7a-c7456" event={"ID":"b0041b29-3335-44fd-943e-66db9471a953","Type":"ContainerStarted","Data":"6e70638cf04e64c977301ab30248e0216ca527cd7f2e713b147befe8c14c95fa"}
Jul 28 21:16:17.182032 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:17.181996    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99e39a7a-c7456"
Jul 28 21:16:17.255435 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:17.255402    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0041b29-3335-44fd-943e-66db9471a953-maas-api-service-ca\") pod \"b0041b29-3335-44fd-943e-66db9471a953\" (UID: \"b0041b29-3335-44fd-943e-66db9471a953\") "
Jul 28 21:16:17.255784 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:17.255753    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0041b29-3335-44fd-943e-66db9471a953-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b0041b29-3335-44fd-943e-66db9471a953" (UID: "b0041b29-3335-44fd-943e-66db9471a953"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:16:17.356727 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:17.356699    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0041b29-3335-44fd-943e-66db9471a953-maas-api-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:16:18.060285 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:18.060253    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99e39a7a-c7456"
Jul 28 21:16:18.060285 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:18.060269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99e39a7a-c7456" event={"ID":"b0041b29-3335-44fd-943e-66db9471a953","Type":"ContainerDied","Data":"6e70638cf04e64c977301ab30248e0216ca527cd7f2e713b147befe8c14c95fa"}
Jul 28 21:16:18.060505 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:18.060328    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e70638cf04e64c977301ab30248e0216ca527cd7f2e713b147befe8c14c95fa"
Jul 28 21:16:20.519178 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:20.519135    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs"]
Jul 28 21:16:20.519680 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:20.519386    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs" podUID="ee977f22-a286-4d3b-bd86-4a12fa64908d" containerName="payload-processing" containerID="cri-o://b6add0cbb7c643867717a9b2703831ae38fd389162a12198433af1e235eddd81" gracePeriod=30
Jul 28 21:16:20.524971 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:20.524944    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4"]
Jul 28 21:16:20.525205 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:20.525177    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4" podUID="08d80d48-a073-46a3-808a-51327ae61595" containerName="payload-pre-processing" containerID="cri-o://cb44edda4810cfd793f3b4ae277e6dbe119eb8242a67528b8168d84945ae22f1" gracePeriod=30
Jul 28 21:16:20.678121 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:20.678084    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99e39a7a-c7456"]
Jul 28 21:16:20.680811 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:20.680784    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99e39a7a-c7456"]
Jul 28 21:16:20.795389 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:20.795359    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs"
Jul 28 21:16:20.799212 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:20.799191    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4"
Jul 28 21:16:20.882727 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:20.882700    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08d80d48-a073-46a3-808a-51327ae61595-plugins-config-volume\") pod \"08d80d48-a073-46a3-808a-51327ae61595\" (UID: \"08d80d48-a073-46a3-808a-51327ae61595\") "
Jul 28 21:16:20.882916 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:20.882774    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlmrm\" (UniqueName: \"kubernetes.io/projected/08d80d48-a073-46a3-808a-51327ae61595-kube-api-access-nlmrm\") pod \"08d80d48-a073-46a3-808a-51327ae61595\" (UID: \"08d80d48-a073-46a3-808a-51327ae61595\") "
Jul 28 21:16:20.882916 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:20.882806    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6994\" (UniqueName: \"kubernetes.io/projected/ee977f22-a286-4d3b-bd86-4a12fa64908d-kube-api-access-l6994\") pod \"ee977f22-a286-4d3b-bd86-4a12fa64908d\" (UID: \"ee977f22-a286-4d3b-bd86-4a12fa64908d\") "
Jul 28 21:16:20.882916 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:20.882825    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee977f22-a286-4d3b-bd86-4a12fa64908d-plugins-config-volume\") pod \"ee977f22-a286-4d3b-bd86-4a12fa64908d\" (UID: \"ee977f22-a286-4d3b-bd86-4a12fa64908d\") "
Jul 28 21:16:20.883214 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:20.883183    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08d80d48-a073-46a3-808a-51327ae61595-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "08d80d48-a073-46a3-808a-51327ae61595" (UID: "08d80d48-a073-46a3-808a-51327ae61595"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:16:20.883392 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:20.883295    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee977f22-a286-4d3b-bd86-4a12fa64908d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ee977f22-a286-4d3b-bd86-4a12fa64908d" (UID: "ee977f22-a286-4d3b-bd86-4a12fa64908d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:16:20.885003 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:20.884982    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d80d48-a073-46a3-808a-51327ae61595-kube-api-access-nlmrm" (OuterVolumeSpecName: "kube-api-access-nlmrm") pod "08d80d48-a073-46a3-808a-51327ae61595" (UID: "08d80d48-a073-46a3-808a-51327ae61595"). InnerVolumeSpecName "kube-api-access-nlmrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:16:20.885088 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:20.884981    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee977f22-a286-4d3b-bd86-4a12fa64908d-kube-api-access-l6994" (OuterVolumeSpecName: "kube-api-access-l6994") pod "ee977f22-a286-4d3b-bd86-4a12fa64908d" (UID: "ee977f22-a286-4d3b-bd86-4a12fa64908d"). InnerVolumeSpecName "kube-api-access-l6994". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:16:20.983373 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:20.983340    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlmrm\" (UniqueName: \"kubernetes.io/projected/08d80d48-a073-46a3-808a-51327ae61595-kube-api-access-nlmrm\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:16:20.983373 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:20.983370    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6994\" (UniqueName: \"kubernetes.io/projected/ee977f22-a286-4d3b-bd86-4a12fa64908d-kube-api-access-l6994\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:16:20.983373 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:20.983380    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee977f22-a286-4d3b-bd86-4a12fa64908d-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:16:20.983598 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:20.983388    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08d80d48-a073-46a3-808a-51327ae61595-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:16:21.081454 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.081360    2565 generic.go:356] "Generic (PLEG): container finished" podID="ee977f22-a286-4d3b-bd86-4a12fa64908d" containerID="b6add0cbb7c643867717a9b2703831ae38fd389162a12198433af1e235eddd81" exitCode=0
Jul 28 21:16:21.081454 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.081401    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs"
Jul 28 21:16:21.081718 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.081453    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs" event={"ID":"ee977f22-a286-4d3b-bd86-4a12fa64908d","Type":"ContainerDied","Data":"b6add0cbb7c643867717a9b2703831ae38fd389162a12198433af1e235eddd81"}
Jul 28 21:16:21.081718 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.081545    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs" event={"ID":"ee977f22-a286-4d3b-bd86-4a12fa64908d","Type":"ContainerDied","Data":"c97a81494ff0e530e30bb776cb0fb7cbfa01c0b2ec063abbb43871bc5d516d34"}
Jul 28 21:16:21.081718 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.081578    2565 scope.go:117] "RemoveContainer" containerID="b6add0cbb7c643867717a9b2703831ae38fd389162a12198433af1e235eddd81"
Jul 28 21:16:21.082799 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.082775    2565 generic.go:356] "Generic (PLEG): container finished" podID="08d80d48-a073-46a3-808a-51327ae61595" containerID="cb44edda4810cfd793f3b4ae277e6dbe119eb8242a67528b8168d84945ae22f1" exitCode=0
Jul 28 21:16:21.082902 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.082807    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4"
Jul 28 21:16:21.082902 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.082816    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4" event={"ID":"08d80d48-a073-46a3-808a-51327ae61595","Type":"ContainerDied","Data":"cb44edda4810cfd793f3b4ae277e6dbe119eb8242a67528b8168d84945ae22f1"}
Jul 28 21:16:21.082902 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.082848    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4" event={"ID":"08d80d48-a073-46a3-808a-51327ae61595","Type":"ContainerDied","Data":"a07e0c3410afd08ad9bdfe268f9450f67fc149cb26ad1314446a360819f5604a"}
Jul 28 21:16:21.092044 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.092026    2565 scope.go:117] "RemoveContainer" containerID="b6add0cbb7c643867717a9b2703831ae38fd389162a12198433af1e235eddd81"
Jul 28 21:16:21.092296 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:16:21.092263    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6add0cbb7c643867717a9b2703831ae38fd389162a12198433af1e235eddd81\": container with ID starting with b6add0cbb7c643867717a9b2703831ae38fd389162a12198433af1e235eddd81 not found: ID does not exist" containerID="b6add0cbb7c643867717a9b2703831ae38fd389162a12198433af1e235eddd81"
Jul 28 21:16:21.092419 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.092300    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6add0cbb7c643867717a9b2703831ae38fd389162a12198433af1e235eddd81"} err="failed to get container status \"b6add0cbb7c643867717a9b2703831ae38fd389162a12198433af1e235eddd81\": rpc error: code = NotFound desc = could not find container \"b6add0cbb7c643867717a9b2703831ae38fd389162a12198433af1e235eddd81\": container with ID starting with b6add0cbb7c643867717a9b2703831ae38fd389162a12198433af1e235eddd81 not found: ID does not exist"
Jul 28 21:16:21.092419 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.092318    2565 scope.go:117] "RemoveContainer" containerID="cb44edda4810cfd793f3b4ae277e6dbe119eb8242a67528b8168d84945ae22f1"
Jul 28 21:16:21.100611 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.100594    2565 scope.go:117] "RemoveContainer" containerID="cb44edda4810cfd793f3b4ae277e6dbe119eb8242a67528b8168d84945ae22f1"
Jul 28 21:16:21.100889 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:16:21.100870    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb44edda4810cfd793f3b4ae277e6dbe119eb8242a67528b8168d84945ae22f1\": container with ID starting with cb44edda4810cfd793f3b4ae277e6dbe119eb8242a67528b8168d84945ae22f1 not found: ID does not exist" containerID="cb44edda4810cfd793f3b4ae277e6dbe119eb8242a67528b8168d84945ae22f1"
Jul 28 21:16:21.100968 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.100895    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb44edda4810cfd793f3b4ae277e6dbe119eb8242a67528b8168d84945ae22f1"} err="failed to get container status \"cb44edda4810cfd793f3b4ae277e6dbe119eb8242a67528b8168d84945ae22f1\": rpc error: code = NotFound desc = could not find container \"cb44edda4810cfd793f3b4ae277e6dbe119eb8242a67528b8168d84945ae22f1\": container with ID starting with cb44edda4810cfd793f3b4ae277e6dbe119eb8242a67528b8168d84945ae22f1 not found: ID does not exist"
Jul 28 21:16:21.105348 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.105326    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4"]
Jul 28 21:16:21.111339 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.111318    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-99e39a7a-7c9bfbb484-46xz4"]
Jul 28 21:16:21.119140 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.119118    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs"]
Jul 28 21:16:21.122909 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.122889    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-99e39a7a-7445f66654-pz5gs"]
Jul 28 21:16:21.331612 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.331528    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08d80d48-a073-46a3-808a-51327ae61595" path="/var/lib/kubelet/pods/08d80d48-a073-46a3-808a-51327ae61595/volumes"
Jul 28 21:16:21.331911 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.331898    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0041b29-3335-44fd-943e-66db9471a953" path="/var/lib/kubelet/pods/b0041b29-3335-44fd-943e-66db9471a953/volumes"
Jul 28 21:16:21.332200 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:21.332188    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee977f22-a286-4d3b-bd86-4a12fa64908d" path="/var/lib/kubelet/pods/ee977f22-a286-4d3b-bd86-4a12fa64908d/volumes"
Jul 28 21:16:24.369792 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:16:24.369758    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:17:47.565836 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:17:47.565732    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:17:59.866953 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:17:59.866914    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:18:08.859054 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:08.859015    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:18:17.725643 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.725596    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj"]
Jul 28 21:18:17.726064 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.725938    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee977f22-a286-4d3b-bd86-4a12fa64908d" containerName="payload-processing"
Jul 28 21:18:17.726064 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.725950    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee977f22-a286-4d3b-bd86-4a12fa64908d" containerName="payload-processing"
Jul 28 21:18:17.726064 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.725971    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08d80d48-a073-46a3-808a-51327ae61595" containerName="payload-pre-processing"
Jul 28 21:18:17.726064 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.725977    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="08d80d48-a073-46a3-808a-51327ae61595" containerName="payload-pre-processing"
Jul 28 21:18:17.726064 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.725984    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0041b29-3335-44fd-943e-66db9471a953" containerName="revoke-keys"
Jul 28 21:18:17.726064 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.725989    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0041b29-3335-44fd-943e-66db9471a953" containerName="revoke-keys"
Jul 28 21:18:17.726064 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.726041    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee977f22-a286-4d3b-bd86-4a12fa64908d" containerName="payload-processing"
Jul 28 21:18:17.726064 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.726051    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="08d80d48-a073-46a3-808a-51327ae61595" containerName="payload-pre-processing"
Jul 28 21:18:17.726064 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.726058    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0041b29-3335-44fd-943e-66db9471a953" containerName="revoke-keys"
Jul 28 21:18:17.729103 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.729088    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj"
Jul 28 21:18:17.731378 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.731352    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7c69f006\""
Jul 28 21:18:17.731532 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.731489    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7c69f006-dockercfg-bc8f9\""
Jul 28 21:18:17.740163 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.740139    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj"]
Jul 28 21:18:17.752072 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.752036    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699"]
Jul 28 21:18:17.755451 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.755430    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699"
Jul 28 21:18:17.765340 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.765314    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699"]
Jul 28 21:18:17.855656 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.855614    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13166c0f-f944-4325-b9fb-8e889462be00-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj\" (UID: \"13166c0f-f944-4325-b9fb-8e889462be00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj"
Jul 28 21:18:17.855821 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.855670    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/599d8d17-1b69-40ae-91c3-95b9821a5188-plugins-config-volume\") pod \"payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699\" (UID: \"599d8d17-1b69-40ae-91c3-95b9821a5188\") " pod="openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699"
Jul 28 21:18:17.855821 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.855710    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7plw\" (UniqueName: \"kubernetes.io/projected/599d8d17-1b69-40ae-91c3-95b9821a5188-kube-api-access-g7plw\") pod \"payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699\" (UID: \"599d8d17-1b69-40ae-91c3-95b9821a5188\") " pod="openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699"
Jul 28 21:18:17.855821 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.855741    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt2md\" (UniqueName: \"kubernetes.io/projected/13166c0f-f944-4325-b9fb-8e889462be00-kube-api-access-vt2md\") pod \"payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj\" (UID: \"13166c0f-f944-4325-b9fb-8e889462be00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj"
Jul 28 21:18:17.956979 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.956941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13166c0f-f944-4325-b9fb-8e889462be00-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj\" (UID: \"13166c0f-f944-4325-b9fb-8e889462be00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj"
Jul 28 21:18:17.957177 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.956994    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/599d8d17-1b69-40ae-91c3-95b9821a5188-plugins-config-volume\") pod \"payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699\" (UID: \"599d8d17-1b69-40ae-91c3-95b9821a5188\") " pod="openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699"
Jul 28 21:18:17.957177 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.957036    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7plw\" (UniqueName: \"kubernetes.io/projected/599d8d17-1b69-40ae-91c3-95b9821a5188-kube-api-access-g7plw\") pod \"payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699\" (UID: \"599d8d17-1b69-40ae-91c3-95b9821a5188\") " pod="openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699"
Jul 28 21:18:17.957177 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.957078    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt2md\" (UniqueName: \"kubernetes.io/projected/13166c0f-f944-4325-b9fb-8e889462be00-kube-api-access-vt2md\") pod \"payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj\" (UID: \"13166c0f-f944-4325-b9fb-8e889462be00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj"
Jul 28 21:18:17.957722 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.957701    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13166c0f-f944-4325-b9fb-8e889462be00-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj\" (UID: \"13166c0f-f944-4325-b9fb-8e889462be00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj"
Jul 28 21:18:17.957823 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.957805    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/599d8d17-1b69-40ae-91c3-95b9821a5188-plugins-config-volume\") pod \"payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699\" (UID: \"599d8d17-1b69-40ae-91c3-95b9821a5188\") " pod="openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699"
Jul 28 21:18:17.967871 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.967854    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt2md\" (UniqueName: \"kubernetes.io/projected/13166c0f-f944-4325-b9fb-8e889462be00-kube-api-access-vt2md\") pod \"payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj\" (UID: \"13166c0f-f944-4325-b9fb-8e889462be00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj"
Jul 28 21:18:17.967933 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:17.967913    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7plw\" (UniqueName: \"kubernetes.io/projected/599d8d17-1b69-40ae-91c3-95b9821a5188-kube-api-access-g7plw\") pod \"payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699\" (UID: \"599d8d17-1b69-40ae-91c3-95b9821a5188\") " pod="openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699"
Jul 28 21:18:18.040476 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:18.040368    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj"
Jul 28 21:18:18.066551 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:18.066522    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699"
Jul 28 21:18:18.188506 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:18.188471    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj"]
Jul 28 21:18:18.190902 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:18:18.190866    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13166c0f_f944_4325_b9fb_8e889462be00.slice/crio-5f80123d0c75ff7e7878317a63b328e4d795d4099cd8e6bd40d2dbf190b2353d WatchSource:0}: Error finding container 5f80123d0c75ff7e7878317a63b328e4d795d4099cd8e6bd40d2dbf190b2353d: Status 404 returned error can't find the container with id 5f80123d0c75ff7e7878317a63b328e4d795d4099cd8e6bd40d2dbf190b2353d
Jul 28 21:18:18.221336 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:18.221302    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699"]
Jul 28 21:18:18.224527 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:18:18.224504    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599d8d17_1b69_40ae_91c3_95b9821a5188.slice/crio-10765cd6bfc2968381ea67aa239c91b964f558ea34905430392d6746bda1e0c7 WatchSource:0}: Error finding container 10765cd6bfc2968381ea67aa239c91b964f558ea34905430392d6746bda1e0c7: Status 404 returned error can't find the container with id 10765cd6bfc2968381ea67aa239c91b964f558ea34905430392d6746bda1e0c7
Jul 28 21:18:18.718981 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:18.718935    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj" event={"ID":"13166c0f-f944-4325-b9fb-8e889462be00","Type":"ContainerStarted","Data":"163ed25aef4ba212d61b2bec6cd88edb09828727ccb84518201e276deb698e10"}
Jul 28 21:18:18.719188 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:18.719008    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj"
Jul 28 21:18:18.719188 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:18.719027    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj" event={"ID":"13166c0f-f944-4325-b9fb-8e889462be00","Type":"ContainerStarted","Data":"5f80123d0c75ff7e7878317a63b328e4d795d4099cd8e6bd40d2dbf190b2353d"}
Jul 28 21:18:18.720451 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:18.720424    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699" event={"ID":"599d8d17-1b69-40ae-91c3-95b9821a5188","Type":"ContainerStarted","Data":"b69fe69c8a2fc5c185d933e81234ec5256c75c2111746a870cdf8604cb334e7e"}
Jul 28 21:18:18.720564 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:18.720466    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699" event={"ID":"599d8d17-1b69-40ae-91c3-95b9821a5188","Type":"ContainerStarted","Data":"10765cd6bfc2968381ea67aa239c91b964f558ea34905430392d6746bda1e0c7"}
Jul 28 21:18:18.720564 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:18.720491    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699"
Jul 28 21:18:18.735230 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:18.735177    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj" podStartSLOduration=1.735160963 podStartE2EDuration="1.735160963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:18:18.733355722 +0000 UTC m=+1853.957698708" watchObservedRunningTime="2026-07-28 21:18:18.735160963 +0000 UTC m=+1853.959503948"
Jul 28 21:18:18.750362 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:18.750311    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699" podStartSLOduration=1.750293181 podStartE2EDuration="1.750293181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:18:18.747826386 +0000 UTC m=+1853.972169371" watchObservedRunningTime="2026-07-28 21:18:18.750293181 +0000 UTC m=+1853.974636232"
Jul 28 21:18:23.455542 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:23.455506    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:18:29.728457 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:29.728427    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699"
Jul 28 21:18:29.728929 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:29.728496    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj"
Jul 28 21:18:49.222729 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:49.222623    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c69f006-5smwp"]
Jul 28 21:18:49.226477 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:49.226454    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c69f006-5smwp"
Jul 28 21:18:49.228504 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:49.228484    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 21:18:49.228613 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:49.228525    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w22xz\""
Jul 28 21:18:49.235683 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:49.235656    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c69f006-5smwp"]
Jul 28 21:18:49.315502 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:49.315470    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/139423d3-162c-4091-992e-585b871d1eda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7c69f006-5smwp\" (UID: \"139423d3-162c-4091-992e-585b871d1eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c69f006-5smwp"
Jul 28 21:18:49.416417 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:49.416383    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/139423d3-162c-4091-992e-585b871d1eda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7c69f006-5smwp\" (UID: \"139423d3-162c-4091-992e-585b871d1eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c69f006-5smwp"
Jul 28 21:18:49.417051 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:49.417033    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/139423d3-162c-4091-992e-585b871d1eda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7c69f006-5smwp\" (UID: \"139423d3-162c-4091-992e-585b871d1eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c69f006-5smwp"
Jul 28 21:18:49.538015 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:49.537929    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c69f006-5smwp"
Jul 28 21:18:49.676083 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:49.675938    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c69f006-5smwp"]
Jul 28 21:18:49.678408 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:18:49.678367    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139423d3_162c_4091_992e_585b871d1eda.slice/crio-841936ace3a191cde536c69c97cd5ecc59d0900dfca83718a86b341e86eb2676 WatchSource:0}: Error finding container 841936ace3a191cde536c69c97cd5ecc59d0900dfca83718a86b341e86eb2676: Status 404 returned error can't find the container with id 841936ace3a191cde536c69c97cd5ecc59d0900dfca83718a86b341e86eb2676
Jul 28 21:18:49.887328 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:49.887295    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c69f006-5smwp" event={"ID":"139423d3-162c-4091-992e-585b871d1eda","Type":"ContainerStarted","Data":"841936ace3a191cde536c69c97cd5ecc59d0900dfca83718a86b341e86eb2676"}
Jul 28 21:18:50.893828 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:50.893797    2565 generic.go:356] "Generic (PLEG): container finished" podID="139423d3-162c-4091-992e-585b871d1eda" containerID="542834e467820cdfc1e2e3d9b6f57dee81109ae6d530a85c725285baae1caa0d" exitCode=0
Jul 28 21:18:50.894286 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:50.893880    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c69f006-5smwp" event={"ID":"139423d3-162c-4091-992e-585b871d1eda","Type":"ContainerDied","Data":"542834e467820cdfc1e2e3d9b6f57dee81109ae6d530a85c725285baae1caa0d"}
Jul 28 21:18:52.028430 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:52.028405    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c69f006-5smwp"
Jul 28 21:18:52.136713 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:52.136679    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/139423d3-162c-4091-992e-585b871d1eda-maas-api-service-ca\") pod \"139423d3-162c-4091-992e-585b871d1eda\" (UID: \"139423d3-162c-4091-992e-585b871d1eda\") "
Jul 28 21:18:52.137045 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:52.137022    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/139423d3-162c-4091-992e-585b871d1eda-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "139423d3-162c-4091-992e-585b871d1eda" (UID: "139423d3-162c-4091-992e-585b871d1eda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:18:52.237254 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:52.237212    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/139423d3-162c-4091-992e-585b871d1eda-maas-api-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:18:52.906300 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:52.906270    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c69f006-5smwp"
Jul 28 21:18:52.906478 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:52.906273    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c69f006-5smwp" event={"ID":"139423d3-162c-4091-992e-585b871d1eda","Type":"ContainerDied","Data":"841936ace3a191cde536c69c97cd5ecc59d0900dfca83718a86b341e86eb2676"}
Jul 28 21:18:52.906478 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:52.906395    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="841936ace3a191cde536c69c97cd5ecc59d0900dfca83718a86b341e86eb2676"
Jul 28 21:18:59.701439 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:59.701403    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699"]
Jul 28 21:18:59.702026 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:59.701828    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699" podUID="599d8d17-1b69-40ae-91c3-95b9821a5188" containerName="payload-processing" containerID="cri-o://b69fe69c8a2fc5c185d933e81234ec5256c75c2111746a870cdf8604cb334e7e" gracePeriod=30
Jul 28 21:18:59.707730 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:59.707705    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj"]
Jul 28 21:18:59.708278 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:59.708072    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj" podUID="13166c0f-f944-4325-b9fb-8e889462be00" containerName="payload-pre-processing" containerID="cri-o://163ed25aef4ba212d61b2bec6cd88edb09828727ccb84518201e276deb698e10" gracePeriod=30
Jul 28 21:18:59.726937 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:59.726907    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj" podUID="13166c0f-f944-4325-b9fb-8e889462be00" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.53:9004: connect: connection refused"
Jul 28 21:18:59.727052 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:59.726907    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699" podUID="599d8d17-1b69-40ae-91c3-95b9821a5188" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.54:9004: connect: connection refused"
Jul 28 21:18:59.854974 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:59.854928    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c69f006-5smwp"]
Jul 28 21:18:59.857499 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:59.857462    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c69f006-5smwp"]
Jul 28 21:18:59.946088 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:59.946057    2565 generic.go:356] "Generic (PLEG): container finished" podID="599d8d17-1b69-40ae-91c3-95b9821a5188" containerID="b69fe69c8a2fc5c185d933e81234ec5256c75c2111746a870cdf8604cb334e7e" exitCode=0
Jul 28 21:18:59.946262 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:59.946134    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699" event={"ID":"599d8d17-1b69-40ae-91c3-95b9821a5188","Type":"ContainerDied","Data":"b69fe69c8a2fc5c185d933e81234ec5256c75c2111746a870cdf8604cb334e7e"}
Jul 28 21:18:59.947924 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:59.947902    2565 generic.go:356] "Generic (PLEG): container finished" podID="13166c0f-f944-4325-b9fb-8e889462be00" containerID="163ed25aef4ba212d61b2bec6cd88edb09828727ccb84518201e276deb698e10" exitCode=0
Jul 28 21:18:59.948038 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:59.947934    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj" event={"ID":"13166c0f-f944-4325-b9fb-8e889462be00","Type":"ContainerDied","Data":"163ed25aef4ba212d61b2bec6cd88edb09828727ccb84518201e276deb698e10"}
Jul 28 21:18:59.985619 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:59.985594    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699"
Jul 28 21:18:59.988363 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:18:59.988346    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj"
Jul 28 21:19:00.109768 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.109730    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7plw\" (UniqueName: \"kubernetes.io/projected/599d8d17-1b69-40ae-91c3-95b9821a5188-kube-api-access-g7plw\") pod \"599d8d17-1b69-40ae-91c3-95b9821a5188\" (UID: \"599d8d17-1b69-40ae-91c3-95b9821a5188\") "
Jul 28 21:19:00.109975 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.109799    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/599d8d17-1b69-40ae-91c3-95b9821a5188-plugins-config-volume\") pod \"599d8d17-1b69-40ae-91c3-95b9821a5188\" (UID: \"599d8d17-1b69-40ae-91c3-95b9821a5188\") "
Jul 28 21:19:00.109975 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.109838    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vt2md\" (UniqueName: \"kubernetes.io/projected/13166c0f-f944-4325-b9fb-8e889462be00-kube-api-access-vt2md\") pod \"13166c0f-f944-4325-b9fb-8e889462be00\" (UID: \"13166c0f-f944-4325-b9fb-8e889462be00\") "
Jul 28 21:19:00.109975 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.109892    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13166c0f-f944-4325-b9fb-8e889462be00-plugins-config-volume\") pod \"13166c0f-f944-4325-b9fb-8e889462be00\" (UID: \"13166c0f-f944-4325-b9fb-8e889462be00\") "
Jul 28 21:19:00.110255 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.110232    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/599d8d17-1b69-40ae-91c3-95b9821a5188-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "599d8d17-1b69-40ae-91c3-95b9821a5188" (UID: "599d8d17-1b69-40ae-91c3-95b9821a5188"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:19:00.110317 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.110260    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13166c0f-f944-4325-b9fb-8e889462be00-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13166c0f-f944-4325-b9fb-8e889462be00" (UID: "13166c0f-f944-4325-b9fb-8e889462be00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:19:00.111978 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.111956    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13166c0f-f944-4325-b9fb-8e889462be00-kube-api-access-vt2md" (OuterVolumeSpecName: "kube-api-access-vt2md") pod "13166c0f-f944-4325-b9fb-8e889462be00" (UID: "13166c0f-f944-4325-b9fb-8e889462be00"). InnerVolumeSpecName "kube-api-access-vt2md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:19:00.112044 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.111981    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599d8d17-1b69-40ae-91c3-95b9821a5188-kube-api-access-g7plw" (OuterVolumeSpecName: "kube-api-access-g7plw") pod "599d8d17-1b69-40ae-91c3-95b9821a5188" (UID: "599d8d17-1b69-40ae-91c3-95b9821a5188"). InnerVolumeSpecName "kube-api-access-g7plw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:19:00.210678 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.210557    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt2md\" (UniqueName: \"kubernetes.io/projected/13166c0f-f944-4325-b9fb-8e889462be00-kube-api-access-vt2md\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:19:00.210678 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.210593    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13166c0f-f944-4325-b9fb-8e889462be00-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:19:00.210678 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.210605    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7plw\" (UniqueName: \"kubernetes.io/projected/599d8d17-1b69-40ae-91c3-95b9821a5188-kube-api-access-g7plw\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:19:00.210678 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.210616    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/599d8d17-1b69-40ae-91c3-95b9821a5188-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:19:00.954724 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.954689    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699"
Jul 28 21:19:00.955200 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.954743    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699" event={"ID":"599d8d17-1b69-40ae-91c3-95b9821a5188","Type":"ContainerDied","Data":"10765cd6bfc2968381ea67aa239c91b964f558ea34905430392d6746bda1e0c7"}
Jul 28 21:19:00.955200 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.954802    2565 scope.go:117] "RemoveContainer" containerID="b69fe69c8a2fc5c185d933e81234ec5256c75c2111746a870cdf8604cb334e7e"
Jul 28 21:19:00.956279 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.956255    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj" event={"ID":"13166c0f-f944-4325-b9fb-8e889462be00","Type":"ContainerDied","Data":"5f80123d0c75ff7e7878317a63b328e4d795d4099cd8e6bd40d2dbf190b2353d"}
Jul 28 21:19:00.956391 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.956272    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj"
Jul 28 21:19:00.964075 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.964051    2565 scope.go:117] "RemoveContainer" containerID="163ed25aef4ba212d61b2bec6cd88edb09828727ccb84518201e276deb698e10"
Jul 28 21:19:00.976125 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.976098    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699"]
Jul 28 21:19:00.982207 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.982180    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7c69f006-58d9fbc5d6-mx699"]
Jul 28 21:19:00.990849 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.990825    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj"]
Jul 28 21:19:00.996744 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:00.996715    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7c69f006-86f6bc99d4-8bmhj"]
Jul 28 21:19:01.331549 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:01.331456    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13166c0f-f944-4325-b9fb-8e889462be00" path="/var/lib/kubelet/pods/13166c0f-f944-4325-b9fb-8e889462be00/volumes"
Jul 28 21:19:01.331884 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:01.331866    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="139423d3-162c-4091-992e-585b871d1eda" path="/var/lib/kubelet/pods/139423d3-162c-4091-992e-585b871d1eda/volumes"
Jul 28 21:19:01.332237 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:01.332221    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="599d8d17-1b69-40ae-91c3-95b9821a5188" path="/var/lib/kubelet/pods/599d8d17-1b69-40ae-91c3-95b9821a5188/volumes"
Jul 28 21:19:02.459176 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:02.459138    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:19:29.821437 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.821405    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf"]
Jul 28 21:19:29.821863 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.821769    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="599d8d17-1b69-40ae-91c3-95b9821a5188" containerName="payload-processing"
Jul 28 21:19:29.821863 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.821781    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="599d8d17-1b69-40ae-91c3-95b9821a5188" containerName="payload-processing"
Jul 28 21:19:29.821863 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.821789    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="139423d3-162c-4091-992e-585b871d1eda" containerName="revoke-keys"
Jul 28 21:19:29.821863 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.821794    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="139423d3-162c-4091-992e-585b871d1eda" containerName="revoke-keys"
Jul 28 21:19:29.821863 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.821806    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13166c0f-f944-4325-b9fb-8e889462be00" containerName="payload-pre-processing"
Jul 28 21:19:29.821863 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.821812    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="13166c0f-f944-4325-b9fb-8e889462be00" containerName="payload-pre-processing"
Jul 28 21:19:29.822048 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.821875    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="139423d3-162c-4091-992e-585b871d1eda" containerName="revoke-keys"
Jul 28 21:19:29.822048 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.821883    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="13166c0f-f944-4325-b9fb-8e889462be00" containerName="payload-pre-processing"
Jul 28 21:19:29.822048 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.821891    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="599d8d17-1b69-40ae-91c3-95b9821a5188" containerName="payload-processing"
Jul 28 21:19:29.825003 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.824986    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf"
Jul 28 21:19:29.826742 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.826719    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6b0b6c51\""
Jul 28 21:19:29.826864 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.826720    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6b0b6c51-dockercfg-sxxmj\""
Jul 28 21:19:29.833942 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.833911    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf"]
Jul 28 21:19:29.852297 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.852264    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq"]
Jul 28 21:19:29.855829 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.855810    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq"
Jul 28 21:19:29.863335 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.863298    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq"]
Jul 28 21:19:29.974104 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.974039    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f6d4bc-542f-4780-9c74-0ba1f5dd0c25-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq\" (UID: \"05f6d4bc-542f-4780-9c74-0ba1f5dd0c25\") " pod="openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq"
Jul 28 21:19:29.974104 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.974098    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwwlb\" (UniqueName: \"kubernetes.io/projected/05f6d4bc-542f-4780-9c74-0ba1f5dd0c25-kube-api-access-fwwlb\") pod \"payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq\" (UID: \"05f6d4bc-542f-4780-9c74-0ba1f5dd0c25\") " pod="openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq"
Jul 28 21:19:29.974329 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.974137    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hckp9\" (UniqueName: \"kubernetes.io/projected/2bba23e2-eaf8-4b4c-8085-99cc84e46f41-kube-api-access-hckp9\") pod \"payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf\" (UID: \"2bba23e2-eaf8-4b4c-8085-99cc84e46f41\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf"
Jul 28 21:19:29.974329 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:29.974187    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bba23e2-eaf8-4b4c-8085-99cc84e46f41-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf\" (UID: \"2bba23e2-eaf8-4b4c-8085-99cc84e46f41\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf"
Jul 28 21:19:30.075481 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:30.075382    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hckp9\" (UniqueName: \"kubernetes.io/projected/2bba23e2-eaf8-4b4c-8085-99cc84e46f41-kube-api-access-hckp9\") pod \"payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf\" (UID: \"2bba23e2-eaf8-4b4c-8085-99cc84e46f41\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf"
Jul 28 21:19:30.075481 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:30.075438    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bba23e2-eaf8-4b4c-8085-99cc84e46f41-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf\" (UID: \"2bba23e2-eaf8-4b4c-8085-99cc84e46f41\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf"
Jul 28 21:19:30.075481 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:30.075481    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f6d4bc-542f-4780-9c74-0ba1f5dd0c25-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq\" (UID: \"05f6d4bc-542f-4780-9c74-0ba1f5dd0c25\") " pod="openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq"
Jul 28 21:19:30.075856 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:30.075516    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwwlb\" (UniqueName: \"kubernetes.io/projected/05f6d4bc-542f-4780-9c74-0ba1f5dd0c25-kube-api-access-fwwlb\") pod \"payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq\" (UID: \"05f6d4bc-542f-4780-9c74-0ba1f5dd0c25\") " pod="openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq"
Jul 28 21:19:30.076240 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:30.076217    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f6d4bc-542f-4780-9c74-0ba1f5dd0c25-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq\" (UID: \"05f6d4bc-542f-4780-9c74-0ba1f5dd0c25\") " pod="openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq"
Jul 28 21:19:30.076346 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:30.076309    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bba23e2-eaf8-4b4c-8085-99cc84e46f41-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf\" (UID: \"2bba23e2-eaf8-4b4c-8085-99cc84e46f41\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf"
Jul 28 21:19:30.082459 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:30.082437    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hckp9\" (UniqueName: \"kubernetes.io/projected/2bba23e2-eaf8-4b4c-8085-99cc84e46f41-kube-api-access-hckp9\") pod \"payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf\" (UID: \"2bba23e2-eaf8-4b4c-8085-99cc84e46f41\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf"
Jul 28 21:19:30.082547 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:30.082526    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwwlb\" (UniqueName: \"kubernetes.io/projected/05f6d4bc-542f-4780-9c74-0ba1f5dd0c25-kube-api-access-fwwlb\") pod \"payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq\" (UID: \"05f6d4bc-542f-4780-9c74-0ba1f5dd0c25\") " pod="openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq"
Jul 28 21:19:30.136530 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:30.136493    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf"
Jul 28 21:19:30.167762 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:30.167725    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq"
Jul 28 21:19:30.292107 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:30.292069    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf"]
Jul 28 21:19:30.293241 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:19:30.293213    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bba23e2_eaf8_4b4c_8085_99cc84e46f41.slice/crio-ff4186c3d5badf54f48e42a4a57217a87acddbdcc965ef854f7f99f0e75b8788 WatchSource:0}: Error finding container ff4186c3d5badf54f48e42a4a57217a87acddbdcc965ef854f7f99f0e75b8788: Status 404 returned error can't find the container with id ff4186c3d5badf54f48e42a4a57217a87acddbdcc965ef854f7f99f0e75b8788
Jul 28 21:19:30.312792 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:30.312766    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq"]
Jul 28 21:19:30.315008 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:19:30.314983    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f6d4bc_542f_4780_9c74_0ba1f5dd0c25.slice/crio-7888add668bdea55c226ade4d3e0d0923d3a83ae4b27e430f00525756c567e3d WatchSource:0}: Error finding container 7888add668bdea55c226ade4d3e0d0923d3a83ae4b27e430f00525756c567e3d: Status 404 returned error can't find the container with id 7888add668bdea55c226ade4d3e0d0923d3a83ae4b27e430f00525756c567e3d
Jul 28 21:19:31.119816 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:31.119782    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq" event={"ID":"05f6d4bc-542f-4780-9c74-0ba1f5dd0c25","Type":"ContainerStarted","Data":"135256c834b1d326a5a0acb0171424e0078a0331e14bae628231de2e0a05dbc6"}
Jul 28 21:19:31.120368 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:31.119830    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq" event={"ID":"05f6d4bc-542f-4780-9c74-0ba1f5dd0c25","Type":"ContainerStarted","Data":"7888add668bdea55c226ade4d3e0d0923d3a83ae4b27e430f00525756c567e3d"}
Jul 28 21:19:31.120368 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:31.119872    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq"
Jul 28 21:19:31.121286 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:31.121261    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf" event={"ID":"2bba23e2-eaf8-4b4c-8085-99cc84e46f41","Type":"ContainerStarted","Data":"10cf68f64716a8c9cf2349ec1c844d74fed99a16a4cc0db1064e5de8b88fcecd"}
Jul 28 21:19:31.121385 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:31.121308    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf"
Jul 28 21:19:31.121385 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:31.121324    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf" event={"ID":"2bba23e2-eaf8-4b4c-8085-99cc84e46f41","Type":"ContainerStarted","Data":"ff4186c3d5badf54f48e42a4a57217a87acddbdcc965ef854f7f99f0e75b8788"}
Jul 28 21:19:31.134060 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:31.134019    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq" podStartSLOduration=2.1340028699999998 podStartE2EDuration="2.13400287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:19:31.132885031 +0000 UTC m=+1926.357228017" watchObservedRunningTime="2026-07-28 21:19:31.13400287 +0000 UTC m=+1926.358345856"
Jul 28 21:19:31.147958 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:31.147913    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf" podStartSLOduration=2.147893843 podStartE2EDuration="2.147893843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:19:31.147262738 +0000 UTC m=+1926.371605727" watchObservedRunningTime="2026-07-28 21:19:31.147893843 +0000 UTC m=+1926.372236829"
Jul 28 21:19:33.962009 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:33.961962    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:19:42.128812 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:42.128779    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq"
Jul 28 21:19:42.129199 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:19:42.128939    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf"
Jul 28 21:20:21.956047 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:21.955958    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:20:34.254016 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:34.253971    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:20:38.514346 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:38.514306    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b0b6c51-hd74d"]
Jul 28 21:20:38.517882 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:38.517860    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b0b6c51-hd74d"
Jul 28 21:20:38.519751 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:38.519730    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w22xz\""
Jul 28 21:20:38.519861 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:38.519729    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 21:20:38.525982 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:38.525955    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b0b6c51-hd74d"]
Jul 28 21:20:38.558210 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:38.558171    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0828dad-65b3-4567-9ecb-b1c8ae9d1159-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b0b6c51-hd74d\" (UID: \"c0828dad-65b3-4567-9ecb-b1c8ae9d1159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b0b6c51-hd74d"
Jul 28 21:20:38.659399 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:38.659356    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0828dad-65b3-4567-9ecb-b1c8ae9d1159-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b0b6c51-hd74d\" (UID: \"c0828dad-65b3-4567-9ecb-b1c8ae9d1159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b0b6c51-hd74d"
Jul 28 21:20:38.660104 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:38.660083    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0828dad-65b3-4567-9ecb-b1c8ae9d1159-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b0b6c51-hd74d\" (UID: \"c0828dad-65b3-4567-9ecb-b1c8ae9d1159\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b0b6c51-hd74d"
Jul 28 21:20:38.829951 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:38.829858    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b0b6c51-hd74d"
Jul 28 21:20:38.971530 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:38.971496    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b0b6c51-hd74d"]
Jul 28 21:20:38.973232 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:20:38.973208    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0828dad_65b3_4567_9ecb_b1c8ae9d1159.slice/crio-3e0e0e5ae6ad4c736cc4048140d3ea26547a972ba6a68e9da5a4869d642dc4c7 WatchSource:0}: Error finding container 3e0e0e5ae6ad4c736cc4048140d3ea26547a972ba6a68e9da5a4869d642dc4c7: Status 404 returned error can't find the container with id 3e0e0e5ae6ad4c736cc4048140d3ea26547a972ba6a68e9da5a4869d642dc4c7
Jul 28 21:20:39.498846 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:39.498806    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b0b6c51-hd74d" event={"ID":"c0828dad-65b3-4567-9ecb-b1c8ae9d1159","Type":"ContainerStarted","Data":"0c255d87755c6de255df0ba87ba7dafca227fdf28d8bb2fbeb64f4cbf819c798"}
Jul 28 21:20:39.499046 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:39.498862    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b0b6c51-hd74d" event={"ID":"c0828dad-65b3-4567-9ecb-b1c8ae9d1159","Type":"ContainerStarted","Data":"3e0e0e5ae6ad4c736cc4048140d3ea26547a972ba6a68e9da5a4869d642dc4c7"}
Jul 28 21:20:39.516858 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:39.516804    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b0b6c51-hd74d" podStartSLOduration=1.516787391 podStartE2EDuration="1.516787391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:20:39.514436193 +0000 UTC m=+1994.738779179" watchObservedRunningTime="2026-07-28 21:20:39.516787391 +0000 UTC m=+1994.741130377"
Jul 28 21:20:40.505597 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:40.505566    2565 generic.go:356] "Generic (PLEG): container finished" podID="c0828dad-65b3-4567-9ecb-b1c8ae9d1159" containerID="0c255d87755c6de255df0ba87ba7dafca227fdf28d8bb2fbeb64f4cbf819c798" exitCode=0
Jul 28 21:20:40.505798 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:40.505674    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b0b6c51-hd74d" event={"ID":"c0828dad-65b3-4567-9ecb-b1c8ae9d1159","Type":"ContainerDied","Data":"0c255d87755c6de255df0ba87ba7dafca227fdf28d8bb2fbeb64f4cbf819c798"}
Jul 28 21:20:41.645615 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:41.645588    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b0b6c51-hd74d"
Jul 28 21:20:41.689614 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:41.689582    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0828dad-65b3-4567-9ecb-b1c8ae9d1159-maas-api-service-ca\") pod \"c0828dad-65b3-4567-9ecb-b1c8ae9d1159\" (UID: \"c0828dad-65b3-4567-9ecb-b1c8ae9d1159\") "
Jul 28 21:20:41.689965 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:41.689941    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0828dad-65b3-4567-9ecb-b1c8ae9d1159-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c0828dad-65b3-4567-9ecb-b1c8ae9d1159" (UID: "c0828dad-65b3-4567-9ecb-b1c8ae9d1159"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:20:41.791062 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:41.790977    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0828dad-65b3-4567-9ecb-b1c8ae9d1159-maas-api-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:20:42.517662 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:42.517605    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b0b6c51-hd74d"
Jul 28 21:20:42.517824 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:42.517663    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b0b6c51-hd74d" event={"ID":"c0828dad-65b3-4567-9ecb-b1c8ae9d1159","Type":"ContainerDied","Data":"3e0e0e5ae6ad4c736cc4048140d3ea26547a972ba6a68e9da5a4869d642dc4c7"}
Jul 28 21:20:42.517824 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:42.517712    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e0e0e5ae6ad4c736cc4048140d3ea26547a972ba6a68e9da5a4869d642dc4c7"
Jul 28 21:20:52.863402 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:52.863362    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:20:54.171471 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.171429    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq"]
Jul 28 21:20:54.172000 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.171668    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq" podUID="05f6d4bc-542f-4780-9c74-0ba1f5dd0c25" containerName="payload-processing" containerID="cri-o://135256c834b1d326a5a0acb0171424e0078a0331e14bae628231de2e0a05dbc6" gracePeriod=30
Jul 28 21:20:54.193793 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.193760    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf"]
Jul 28 21:20:54.193980 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.193959    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf" podUID="2bba23e2-eaf8-4b4c-8085-99cc84e46f41" containerName="payload-pre-processing" containerID="cri-o://10cf68f64716a8c9cf2349ec1c844d74fed99a16a4cc0db1064e5de8b88fcecd" gracePeriod=30
Jul 28 21:20:54.379926 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.379881    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b0b6c51-hd74d"]
Jul 28 21:20:54.381979 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.381955    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b0b6c51-hd74d"]
Jul 28 21:20:54.467185 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.467156    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq"
Jul 28 21:20:54.470026 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.470007    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf"
Jul 28 21:20:54.584253 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.584218    2565 generic.go:356] "Generic (PLEG): container finished" podID="05f6d4bc-542f-4780-9c74-0ba1f5dd0c25" containerID="135256c834b1d326a5a0acb0171424e0078a0331e14bae628231de2e0a05dbc6" exitCode=0
Jul 28 21:20:54.584465 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.584267    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq"
Jul 28 21:20:54.584465 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.584320    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq" event={"ID":"05f6d4bc-542f-4780-9c74-0ba1f5dd0c25","Type":"ContainerDied","Data":"135256c834b1d326a5a0acb0171424e0078a0331e14bae628231de2e0a05dbc6"}
Jul 28 21:20:54.584465 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.584377    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq" event={"ID":"05f6d4bc-542f-4780-9c74-0ba1f5dd0c25","Type":"ContainerDied","Data":"7888add668bdea55c226ade4d3e0d0923d3a83ae4b27e430f00525756c567e3d"}
Jul 28 21:20:54.584465 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.584412    2565 scope.go:117] "RemoveContainer" containerID="135256c834b1d326a5a0acb0171424e0078a0331e14bae628231de2e0a05dbc6"
Jul 28 21:20:54.585732 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.585706    2565 generic.go:356] "Generic (PLEG): container finished" podID="2bba23e2-eaf8-4b4c-8085-99cc84e46f41" containerID="10cf68f64716a8c9cf2349ec1c844d74fed99a16a4cc0db1064e5de8b88fcecd" exitCode=0
Jul 28 21:20:54.585856 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.585760    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf"
Jul 28 21:20:54.585856 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.585810    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf" event={"ID":"2bba23e2-eaf8-4b4c-8085-99cc84e46f41","Type":"ContainerDied","Data":"10cf68f64716a8c9cf2349ec1c844d74fed99a16a4cc0db1064e5de8b88fcecd"}
Jul 28 21:20:54.585941 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.585856    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf" event={"ID":"2bba23e2-eaf8-4b4c-8085-99cc84e46f41","Type":"ContainerDied","Data":"ff4186c3d5badf54f48e42a4a57217a87acddbdcc965ef854f7f99f0e75b8788"}
Jul 28 21:20:54.593576 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.593558    2565 scope.go:117] "RemoveContainer" containerID="135256c834b1d326a5a0acb0171424e0078a0331e14bae628231de2e0a05dbc6"
Jul 28 21:20:54.593900 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:20:54.593880    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"135256c834b1d326a5a0acb0171424e0078a0331e14bae628231de2e0a05dbc6\": container with ID starting with 135256c834b1d326a5a0acb0171424e0078a0331e14bae628231de2e0a05dbc6 not found: ID does not exist" containerID="135256c834b1d326a5a0acb0171424e0078a0331e14bae628231de2e0a05dbc6"
Jul 28 21:20:54.593967 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.593907    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"135256c834b1d326a5a0acb0171424e0078a0331e14bae628231de2e0a05dbc6"} err="failed to get container status \"135256c834b1d326a5a0acb0171424e0078a0331e14bae628231de2e0a05dbc6\": rpc error: code = NotFound desc = could not find container \"135256c834b1d326a5a0acb0171424e0078a0331e14bae628231de2e0a05dbc6\": container with ID starting with 135256c834b1d326a5a0acb0171424e0078a0331e14bae628231de2e0a05dbc6 not found: ID does not exist"
Jul 28 21:20:54.593967 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.593936    2565 scope.go:117] "RemoveContainer" containerID="10cf68f64716a8c9cf2349ec1c844d74fed99a16a4cc0db1064e5de8b88fcecd"
Jul 28 21:20:54.602179 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.602158    2565 scope.go:117] "RemoveContainer" containerID="10cf68f64716a8c9cf2349ec1c844d74fed99a16a4cc0db1064e5de8b88fcecd"
Jul 28 21:20:54.602442 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:20:54.602424    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10cf68f64716a8c9cf2349ec1c844d74fed99a16a4cc0db1064e5de8b88fcecd\": container with ID starting with 10cf68f64716a8c9cf2349ec1c844d74fed99a16a4cc0db1064e5de8b88fcecd not found: ID does not exist" containerID="10cf68f64716a8c9cf2349ec1c844d74fed99a16a4cc0db1064e5de8b88fcecd"
Jul 28 21:20:54.602490 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.602451    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10cf68f64716a8c9cf2349ec1c844d74fed99a16a4cc0db1064e5de8b88fcecd"} err="failed to get container status \"10cf68f64716a8c9cf2349ec1c844d74fed99a16a4cc0db1064e5de8b88fcecd\": rpc error: code = NotFound desc = could not find container \"10cf68f64716a8c9cf2349ec1c844d74fed99a16a4cc0db1064e5de8b88fcecd\": container with ID starting with 10cf68f64716a8c9cf2349ec1c844d74fed99a16a4cc0db1064e5de8b88fcecd not found: ID does not exist"
Jul 28 21:20:54.605766 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.605746    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bba23e2-eaf8-4b4c-8085-99cc84e46f41-plugins-config-volume\") pod \"2bba23e2-eaf8-4b4c-8085-99cc84e46f41\" (UID: \"2bba23e2-eaf8-4b4c-8085-99cc84e46f41\") "
Jul 28 21:20:54.605836 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.605792    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f6d4bc-542f-4780-9c74-0ba1f5dd0c25-plugins-config-volume\") pod \"05f6d4bc-542f-4780-9c74-0ba1f5dd0c25\" (UID: \"05f6d4bc-542f-4780-9c74-0ba1f5dd0c25\") "
Jul 28 21:20:54.605836 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.605813    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwwlb\" (UniqueName: \"kubernetes.io/projected/05f6d4bc-542f-4780-9c74-0ba1f5dd0c25-kube-api-access-fwwlb\") pod \"05f6d4bc-542f-4780-9c74-0ba1f5dd0c25\" (UID: \"05f6d4bc-542f-4780-9c74-0ba1f5dd0c25\") "
Jul 28 21:20:54.605836 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.605833    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hckp9\" (UniqueName: \"kubernetes.io/projected/2bba23e2-eaf8-4b4c-8085-99cc84e46f41-kube-api-access-hckp9\") pod \"2bba23e2-eaf8-4b4c-8085-99cc84e46f41\" (UID: \"2bba23e2-eaf8-4b4c-8085-99cc84e46f41\") "
Jul 28 21:20:54.606166 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.606142    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bba23e2-eaf8-4b4c-8085-99cc84e46f41-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2bba23e2-eaf8-4b4c-8085-99cc84e46f41" (UID: "2bba23e2-eaf8-4b4c-8085-99cc84e46f41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:20:54.606238 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.606176    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f6d4bc-542f-4780-9c74-0ba1f5dd0c25-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "05f6d4bc-542f-4780-9c74-0ba1f5dd0c25" (UID: "05f6d4bc-542f-4780-9c74-0ba1f5dd0c25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:20:54.607949 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.607922    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f6d4bc-542f-4780-9c74-0ba1f5dd0c25-kube-api-access-fwwlb" (OuterVolumeSpecName: "kube-api-access-fwwlb") pod "05f6d4bc-542f-4780-9c74-0ba1f5dd0c25" (UID: "05f6d4bc-542f-4780-9c74-0ba1f5dd0c25"). InnerVolumeSpecName "kube-api-access-fwwlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:20:54.608052 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.607956    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bba23e2-eaf8-4b4c-8085-99cc84e46f41-kube-api-access-hckp9" (OuterVolumeSpecName: "kube-api-access-hckp9") pod "2bba23e2-eaf8-4b4c-8085-99cc84e46f41" (UID: "2bba23e2-eaf8-4b4c-8085-99cc84e46f41"). InnerVolumeSpecName "kube-api-access-hckp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:20:54.706467 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.706367    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bba23e2-eaf8-4b4c-8085-99cc84e46f41-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:20:54.706467 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.706404    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05f6d4bc-542f-4780-9c74-0ba1f5dd0c25-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:20:54.706467 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.706413    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwwlb\" (UniqueName: \"kubernetes.io/projected/05f6d4bc-542f-4780-9c74-0ba1f5dd0c25-kube-api-access-fwwlb\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:20:54.706467 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.706422    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hckp9\" (UniqueName: \"kubernetes.io/projected/2bba23e2-eaf8-4b4c-8085-99cc84e46f41-kube-api-access-hckp9\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:20:54.909988 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.909948    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq"]
Jul 28 21:20:54.915305 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.915277    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b0b6c51-5c59ff54d6-zhxlq"]
Jul 28 21:20:54.925352 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.925318    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf"]
Jul 28 21:20:54.929625 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:54.929597    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b0b6c51-6f66969f84-mkxlf"]
Jul 28 21:20:55.330981 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:55.330947    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05f6d4bc-542f-4780-9c74-0ba1f5dd0c25" path="/var/lib/kubelet/pods/05f6d4bc-542f-4780-9c74-0ba1f5dd0c25/volumes"
Jul 28 21:20:55.331375 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:55.331275    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bba23e2-eaf8-4b4c-8085-99cc84e46f41" path="/var/lib/kubelet/pods/2bba23e2-eaf8-4b4c-8085-99cc84e46f41/volumes"
Jul 28 21:20:55.331550 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:20:55.331538    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0828dad-65b3-4567-9ecb-b1c8ae9d1159" path="/var/lib/kubelet/pods/c0828dad-65b3-4567-9ecb-b1c8ae9d1159/volumes"
Jul 28 21:21:01.169880 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:21:01.169834    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:21:27.139935 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:21:27.139900    2565 scope.go:117] "RemoveContainer" containerID="e19be05242100e03811f73cacc286e1e78a750ab8f04ac1f27e7fa0df57fda91"
Jul 28 21:21:27.148988 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:21:27.148893    2565 scope.go:117] "RemoveContainer" containerID="ee4f1a976cbee7870dd72ae82b69439223f6a51c1e58ecf0a2f17158961ad12f"
Jul 28 21:21:27.157924 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:21:27.157902    2565 scope.go:117] "RemoveContainer" containerID="fc10835114614445c4d057ba4fdda061b27f0fc8273957c80b7e73eaa71cb0b4"
Jul 28 21:22:04.265564 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:04.265527    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:22:16.569868 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:16.569831    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:22:27.193892 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:27.193861    2565 scope.go:117] "RemoveContainer" containerID="630a361bff191fd764bbc87cfcccbc4a4cb19130f871fb9fe7b69bf2a12dec2c"
Jul 28 21:22:27.202140 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:27.202119    2565 scope.go:117] "RemoveContainer" containerID="d3ad450103f070d8af0f82a64e9a9f95c29cc3389625a11b9ea29b926d8ea10f"
Jul 28 21:22:59.161234 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.161190    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj"]
Jul 28 21:22:59.163822 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.161685    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05f6d4bc-542f-4780-9c74-0ba1f5dd0c25" containerName="payload-processing"
Jul 28 21:22:59.163822 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.161704    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="05f6d4bc-542f-4780-9c74-0ba1f5dd0c25" containerName="payload-processing"
Jul 28 21:22:59.163822 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.161733    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0828dad-65b3-4567-9ecb-b1c8ae9d1159" containerName="revoke-keys"
Jul 28 21:22:59.163822 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.161742    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0828dad-65b3-4567-9ecb-b1c8ae9d1159" containerName="revoke-keys"
Jul 28 21:22:59.163822 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.161759    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bba23e2-eaf8-4b4c-8085-99cc84e46f41" containerName="payload-pre-processing"
Jul 28 21:22:59.163822 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.161765    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bba23e2-eaf8-4b4c-8085-99cc84e46f41" containerName="payload-pre-processing"
Jul 28 21:22:59.163822 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.161827    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="05f6d4bc-542f-4780-9c74-0ba1f5dd0c25" containerName="payload-processing"
Jul 28 21:22:59.163822 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.161835    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0828dad-65b3-4567-9ecb-b1c8ae9d1159" containerName="revoke-keys"
Jul 28 21:22:59.163822 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.161843    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bba23e2-eaf8-4b4c-8085-99cc84e46f41" containerName="payload-pre-processing"
Jul 28 21:22:59.164805 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.164785    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj"
Jul 28 21:22:59.166690 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.166665    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b16ceb-dockercfg-zprmv\""
Jul 28 21:22:59.166799 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.166705    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b16ceb\""
Jul 28 21:22:59.174561 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.174537    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj"]
Jul 28 21:22:59.182956 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.182917    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq9jh\" (UniqueName: \"kubernetes.io/projected/819b169c-a7a0-40ae-b5bf-78641ac7ea91-kube-api-access-lq9jh\") pod \"payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj\" (UID: \"819b169c-a7a0-40ae-b5bf-78641ac7ea91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj"
Jul 28 21:22:59.183096 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.183001    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/819b169c-a7a0-40ae-b5bf-78641ac7ea91-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj\" (UID: \"819b169c-a7a0-40ae-b5bf-78641ac7ea91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj"
Jul 28 21:22:59.193616 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.193585    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf"]
Jul 28 21:22:59.196874 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.196855    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf"
Jul 28 21:22:59.206908 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.206884    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf"]
Jul 28 21:22:59.283520 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.283476    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8fc13a-6ddf-4278-befc-f023d42a5530-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf\" (UID: \"cb8fc13a-6ddf-4278-befc-f023d42a5530\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf"
Jul 28 21:22:59.283712 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.283527    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdscf\" (UniqueName: \"kubernetes.io/projected/cb8fc13a-6ddf-4278-befc-f023d42a5530-kube-api-access-xdscf\") pod \"payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf\" (UID: \"cb8fc13a-6ddf-4278-befc-f023d42a5530\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf"
Jul 28 21:22:59.283712 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.283616    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq9jh\" (UniqueName: \"kubernetes.io/projected/819b169c-a7a0-40ae-b5bf-78641ac7ea91-kube-api-access-lq9jh\") pod \"payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj\" (UID: \"819b169c-a7a0-40ae-b5bf-78641ac7ea91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj"
Jul 28 21:22:59.283712 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.283679    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/819b169c-a7a0-40ae-b5bf-78641ac7ea91-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj\" (UID: \"819b169c-a7a0-40ae-b5bf-78641ac7ea91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj"
Jul 28 21:22:59.284346 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.284326    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/819b169c-a7a0-40ae-b5bf-78641ac7ea91-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj\" (UID: \"819b169c-a7a0-40ae-b5bf-78641ac7ea91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj"
Jul 28 21:22:59.291372 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.291338    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq9jh\" (UniqueName: \"kubernetes.io/projected/819b169c-a7a0-40ae-b5bf-78641ac7ea91-kube-api-access-lq9jh\") pod \"payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj\" (UID: \"819b169c-a7a0-40ae-b5bf-78641ac7ea91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj"
Jul 28 21:22:59.384098 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.384057    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8fc13a-6ddf-4278-befc-f023d42a5530-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf\" (UID: \"cb8fc13a-6ddf-4278-befc-f023d42a5530\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf"
Jul 28 21:22:59.384098 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.384095    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdscf\" (UniqueName: \"kubernetes.io/projected/cb8fc13a-6ddf-4278-befc-f023d42a5530-kube-api-access-xdscf\") pod \"payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf\" (UID: \"cb8fc13a-6ddf-4278-befc-f023d42a5530\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf"
Jul 28 21:22:59.384852 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.384832    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8fc13a-6ddf-4278-befc-f023d42a5530-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf\" (UID: \"cb8fc13a-6ddf-4278-befc-f023d42a5530\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf"
Jul 28 21:22:59.391630 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.391599    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdscf\" (UniqueName: \"kubernetes.io/projected/cb8fc13a-6ddf-4278-befc-f023d42a5530-kube-api-access-xdscf\") pod \"payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf\" (UID: \"cb8fc13a-6ddf-4278-befc-f023d42a5530\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf"
Jul 28 21:22:59.476215 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.476177    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj"
Jul 28 21:22:59.507478 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.507441    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf"
Jul 28 21:22:59.624595 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.624336    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj"]
Jul 28 21:22:59.630033 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:22:59.629978    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819b169c_a7a0_40ae_b5bf_78641ac7ea91.slice/crio-cc99426010602ea8cd16ccf0d1c70b84bc32431d7c98c4234a146da163801de9 WatchSource:0}: Error finding container cc99426010602ea8cd16ccf0d1c70b84bc32431d7c98c4234a146da163801de9: Status 404 returned error can't find the container with id cc99426010602ea8cd16ccf0d1c70b84bc32431d7c98c4234a146da163801de9
Jul 28 21:22:59.654947 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:22:59.654914    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf"]
Jul 28 21:22:59.662798 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:22:59.662767    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8fc13a_6ddf_4278_befc_f023d42a5530.slice/crio-6e56d8eacaeff33b6297249e860df30782727b45e40da1540634e9401a0df51f WatchSource:0}: Error finding container 6e56d8eacaeff33b6297249e860df30782727b45e40da1540634e9401a0df51f: Status 404 returned error can't find the container with id 6e56d8eacaeff33b6297249e860df30782727b45e40da1540634e9401a0df51f
Jul 28 21:23:00.257936 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:00.257883    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj" event={"ID":"819b169c-a7a0-40ae-b5bf-78641ac7ea91","Type":"ContainerStarted","Data":"a4e7d3428f5d1b320888d3fa1322a309540a16c610291e00e96016293e901451"}
Jul 28 21:23:00.258417 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:00.257953    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj" event={"ID":"819b169c-a7a0-40ae-b5bf-78641ac7ea91","Type":"ContainerStarted","Data":"cc99426010602ea8cd16ccf0d1c70b84bc32431d7c98c4234a146da163801de9"}
Jul 28 21:23:00.258417 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:00.258048    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj"
Jul 28 21:23:00.259457 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:00.259432    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf" event={"ID":"cb8fc13a-6ddf-4278-befc-f023d42a5530","Type":"ContainerStarted","Data":"6889950865e844a42a16b585c36a63a984042d2ed07587a98f4c81f128b50ea3"}
Jul 28 21:23:00.259582 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:00.259472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf" event={"ID":"cb8fc13a-6ddf-4278-befc-f023d42a5530","Type":"ContainerStarted","Data":"6e56d8eacaeff33b6297249e860df30782727b45e40da1540634e9401a0df51f"}
Jul 28 21:23:00.259582 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:00.259521    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf"
Jul 28 21:23:00.275944 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:00.275891    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj" podStartSLOduration=1.275875173 podStartE2EDuration="1.275875173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:23:00.274442273 +0000 UTC m=+2135.498785262" watchObservedRunningTime="2026-07-28 21:23:00.275875173 +0000 UTC m=+2135.500218158"
Jul 28 21:23:00.290767 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:00.290707    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf" podStartSLOduration=1.290687023 podStartE2EDuration="1.290687023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:23:00.289944934 +0000 UTC m=+2135.514287921" watchObservedRunningTime="2026-07-28 21:23:00.290687023 +0000 UTC m=+2135.515030009"
Jul 28 21:23:03.860904 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:03.860871    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:23:11.266823 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:11.266787    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf"
Jul 28 21:23:11.267212 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:11.267080    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj"
Jul 28 21:23:27.171711 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.171666    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7"]
Jul 28 21:23:27.176579 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.176550    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7"
Jul 28 21:23:27.178383 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.178360    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-75f962\""
Jul 28 21:23:27.182691 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.182663    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7"]
Jul 28 21:23:27.219789 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.219747    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr"]
Jul 28 21:23:27.223149 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.223131    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr"
Jul 28 21:23:27.224912 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.224887    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-75f962-dockercfg-qt6mk\""
Jul 28 21:23:27.231816 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.231789    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr"]
Jul 28 21:23:27.304885 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.304851    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3015f78e-a332-4c4c-b538-61ae496d68b1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr\" (UID: \"3015f78e-a332-4c4c-b538-61ae496d68b1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr"
Jul 28 21:23:27.304885 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.304888    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5g5w\" (UniqueName: \"kubernetes.io/projected/3015f78e-a332-4c4c-b538-61ae496d68b1-kube-api-access-c5g5w\") pod \"payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr\" (UID: \"3015f78e-a332-4c4c-b538-61ae496d68b1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr"
Jul 28 21:23:27.305099 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.304961    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-599l9\" (UniqueName: \"kubernetes.io/projected/f2b8fb7f-a7ba-4c72-974b-92403f5b0234-kube-api-access-599l9\") pod \"payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7\" (UID: \"f2b8fb7f-a7ba-4c72-974b-92403f5b0234\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7"
Jul 28 21:23:27.305099 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.304994    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b8fb7f-a7ba-4c72-974b-92403f5b0234-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7\" (UID: \"f2b8fb7f-a7ba-4c72-974b-92403f5b0234\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7"
Jul 28 21:23:27.405883 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.405850    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b8fb7f-a7ba-4c72-974b-92403f5b0234-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7\" (UID: \"f2b8fb7f-a7ba-4c72-974b-92403f5b0234\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7"
Jul 28 21:23:27.406089 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.405950    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3015f78e-a332-4c4c-b538-61ae496d68b1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr\" (UID: \"3015f78e-a332-4c4c-b538-61ae496d68b1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr"
Jul 28 21:23:27.406089 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.405979    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5g5w\" (UniqueName: \"kubernetes.io/projected/3015f78e-a332-4c4c-b538-61ae496d68b1-kube-api-access-c5g5w\") pod \"payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr\" (UID: \"3015f78e-a332-4c4c-b538-61ae496d68b1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr"
Jul 28 21:23:27.406228 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.406161    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-599l9\" (UniqueName: \"kubernetes.io/projected/f2b8fb7f-a7ba-4c72-974b-92403f5b0234-kube-api-access-599l9\") pod \"payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7\" (UID: \"f2b8fb7f-a7ba-4c72-974b-92403f5b0234\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7"
Jul 28 21:23:27.406717 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.406693    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b8fb7f-a7ba-4c72-974b-92403f5b0234-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7\" (UID: \"f2b8fb7f-a7ba-4c72-974b-92403f5b0234\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7"
Jul 28 21:23:27.406717 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.406713    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3015f78e-a332-4c4c-b538-61ae496d68b1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr\" (UID: \"3015f78e-a332-4c4c-b538-61ae496d68b1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr"
Jul 28 21:23:27.413198 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.413175    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-599l9\" (UniqueName: \"kubernetes.io/projected/f2b8fb7f-a7ba-4c72-974b-92403f5b0234-kube-api-access-599l9\") pod \"payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7\" (UID: \"f2b8fb7f-a7ba-4c72-974b-92403f5b0234\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7"
Jul 28 21:23:27.413287 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.413231    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5g5w\" (UniqueName: \"kubernetes.io/projected/3015f78e-a332-4c4c-b538-61ae496d68b1-kube-api-access-c5g5w\") pod \"payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr\" (UID: \"3015f78e-a332-4c4c-b538-61ae496d68b1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr"
Jul 28 21:23:27.489602 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.489565    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7"
Jul 28 21:23:27.535952 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.535914    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr"
Jul 28 21:23:27.638526 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.638502    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7"]
Jul 28 21:23:27.639021 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:23:27.638998    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b8fb7f_a7ba_4c72_974b_92403f5b0234.slice/crio-61b881bf9f4bc988d5633249920cba3c3569b5f0aa347cf0887900d9c90c3d0c WatchSource:0}: Error finding container 61b881bf9f4bc988d5633249920cba3c3569b5f0aa347cf0887900d9c90c3d0c: Status 404 returned error can't find the container with id 61b881bf9f4bc988d5633249920cba3c3569b5f0aa347cf0887900d9c90c3d0c
Jul 28 21:23:27.696888 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:27.696861    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr"]
Jul 28 21:23:27.698106 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:23:27.698078    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3015f78e_a332_4c4c_b538_61ae496d68b1.slice/crio-c02002e23b41344ccc845483af36f8570be974609c6900e46f2e29d90c61c738 WatchSource:0}: Error finding container c02002e23b41344ccc845483af36f8570be974609c6900e46f2e29d90c61c738: Status 404 returned error can't find the container with id c02002e23b41344ccc845483af36f8570be974609c6900e46f2e29d90c61c738
Jul 28 21:23:28.403245 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:28.403206    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr" event={"ID":"3015f78e-a332-4c4c-b538-61ae496d68b1","Type":"ContainerStarted","Data":"cce9e48882a9091514b1eb670015751cf96f05abfc98b8fc8dcd75f467dea44e"}
Jul 28 21:23:28.403788 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:28.403286    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr" event={"ID":"3015f78e-a332-4c4c-b538-61ae496d68b1","Type":"ContainerStarted","Data":"c02002e23b41344ccc845483af36f8570be974609c6900e46f2e29d90c61c738"}
Jul 28 21:23:28.403788 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:28.403323    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr"
Jul 28 21:23:28.404904 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:28.404867    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7" event={"ID":"f2b8fb7f-a7ba-4c72-974b-92403f5b0234","Type":"ContainerStarted","Data":"cee651562ec35faed0f6ed18ea47c40e2bc925a0ea2549c5a84bc3858af8356a"}
Jul 28 21:23:28.405012 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:28.404930    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7"
Jul 28 21:23:28.405012 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:28.404947    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7" event={"ID":"f2b8fb7f-a7ba-4c72-974b-92403f5b0234","Type":"ContainerStarted","Data":"61b881bf9f4bc988d5633249920cba3c3569b5f0aa347cf0887900d9c90c3d0c"}
Jul 28 21:23:28.419477 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:28.419422    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr" podStartSLOduration=1.419402444 podStartE2EDuration="1.419402444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:23:28.416805127 +0000 UTC m=+2163.641148115" watchObservedRunningTime="2026-07-28 21:23:28.419402444 +0000 UTC m=+2163.643745431"
Jul 28 21:23:28.433386 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:28.433343    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7" podStartSLOduration=1.433329166 podStartE2EDuration="1.433329166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:23:28.431231443 +0000 UTC m=+2163.655574430" watchObservedRunningTime="2026-07-28 21:23:28.433329166 +0000 UTC m=+2163.657672151"
Jul 28 21:23:33.166966 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:33.166931    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:23:39.412821 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:39.412788    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr"
Jul 28 21:23:39.413273 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:23:39.412999    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7"
Jul 28 21:24:06.363783 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:06.363745    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:24:28.775491 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.775454    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"]
Jul 28 21:24:28.779071 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.779052    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:28.781550 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.781509    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-75f962\"/\"openshift-service-ca.crt\""
Jul 28 21:24:28.781550 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.781545    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-75f962\"/\"test-model-75f962-kserve-self-signed-certs\""
Jul 28 21:24:28.781731 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.781524    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-75f962\"/\"default-dockercfg-q58v7\""
Jul 28 21:24:28.781731 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.781517    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-75f962\"/\"kube-root-ca.crt\""
Jul 28 21:24:28.788061 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.788036    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"]
Jul 28 21:24:28.832161 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.832127    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh82k\" (UniqueName: \"kubernetes.io/projected/03f68325-2c63-4383-81de-f091e747d029-kube-api-access-hh82k\") pod \"test-model-75f962-kserve-699569c979-mbm47\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") " pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:28.832161 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.832166    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03f68325-2c63-4383-81de-f091e747d029-tls-certs\") pod \"test-model-75f962-kserve-699569c979-mbm47\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") " pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:28.832400 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.832231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-dshm\") pod \"test-model-75f962-kserve-699569c979-mbm47\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") " pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:28.832400 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.832281    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-tmp-dir\") pod \"test-model-75f962-kserve-699569c979-mbm47\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") " pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:28.832400 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.832336    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-model-cache\") pod \"test-model-75f962-kserve-699569c979-mbm47\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") " pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:28.832517 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.832408    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-home\") pod \"test-model-75f962-kserve-699569c979-mbm47\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") " pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:28.933750 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.933710    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-home\") pod \"test-model-75f962-kserve-699569c979-mbm47\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") " pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:28.933946 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.933760    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh82k\" (UniqueName: \"kubernetes.io/projected/03f68325-2c63-4383-81de-f091e747d029-kube-api-access-hh82k\") pod \"test-model-75f962-kserve-699569c979-mbm47\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") " pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:28.933946 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.933793    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03f68325-2c63-4383-81de-f091e747d029-tls-certs\") pod \"test-model-75f962-kserve-699569c979-mbm47\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") " pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:28.933946 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.933835    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-dshm\") pod \"test-model-75f962-kserve-699569c979-mbm47\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") " pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:28.933946 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.933863    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-tmp-dir\") pod \"test-model-75f962-kserve-699569c979-mbm47\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") " pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:28.933946 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.933905    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-model-cache\") pod \"test-model-75f962-kserve-699569c979-mbm47\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") " pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:28.934219 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.934186    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-home\") pod \"test-model-75f962-kserve-699569c979-mbm47\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") " pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:28.934219 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.934210    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-model-cache\") pod \"test-model-75f962-kserve-699569c979-mbm47\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") " pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:28.934330 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.934310    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-tmp-dir\") pod \"test-model-75f962-kserve-699569c979-mbm47\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") " pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:28.936084 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.936065    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-dshm\") pod \"test-model-75f962-kserve-699569c979-mbm47\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") " pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:28.936386 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.936370    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03f68325-2c63-4383-81de-f091e747d029-tls-certs\") pod \"test-model-75f962-kserve-699569c979-mbm47\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") " pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:28.942469 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:28.942448    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh82k\" (UniqueName: \"kubernetes.io/projected/03f68325-2c63-4383-81de-f091e747d029-kube-api-access-hh82k\") pod \"test-model-75f962-kserve-699569c979-mbm47\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") " pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:29.089439 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:29.089352    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:29.226318 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:29.226287    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"]
Jul 28 21:24:29.228564 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:24:29.228530    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03f68325_2c63_4383_81de_f091e747d029.slice/crio-aeae6ad87d25a5ec670f5fb1f7b7d9c605454d246ea2a3494bc7aaf602b23f19 WatchSource:0}: Error finding container aeae6ad87d25a5ec670f5fb1f7b7d9c605454d246ea2a3494bc7aaf602b23f19: Status 404 returned error can't find the container with id aeae6ad87d25a5ec670f5fb1f7b7d9c605454d246ea2a3494bc7aaf602b23f19
Jul 28 21:24:29.230576 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:29.230559    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 21:24:29.747936 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:29.747879    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47" event={"ID":"03f68325-2c63-4383-81de-f091e747d029","Type":"ContainerStarted","Data":"aeae6ad87d25a5ec670f5fb1f7b7d9c605454d246ea2a3494bc7aaf602b23f19"}
Jul 28 21:24:30.755383 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:30.755350    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47" event={"ID":"03f68325-2c63-4383-81de-f091e747d029","Type":"ContainerStarted","Data":"f9086de5ab2ccedc5990bcba7cf2b4376cb4b74a7abdd7edf4b922fdf245ef1d"}
Jul 28 21:24:30.793063 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:30.793021    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47" podStartSLOduration=1.647889101 podStartE2EDuration="2.793006777s" totalImagesPullingTime="1.145117676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:24:30.792155976 +0000 UTC m=+2226.016498963" watchObservedRunningTime="2026-07-28 21:24:30.793006777 +0000 UTC m=+2226.017349774"
Jul 28 21:24:33.160277 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:33.160238    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:24:39.090130 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:39.090091    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:39.090130 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:39.090142    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:39.103177 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:39.103147    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:39.814246 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:39.814218    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:24:45.862211 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:24:45.862176    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:25:21.899528 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:21.899495    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"]
Jul 28 21:25:22.045943 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:22.045899    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b16ceb-9sgs4"]
Jul 28 21:25:22.049433 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:22.049412    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b16ceb-9sgs4"
Jul 28 21:25:22.051006 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:22.050985    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 21:25:22.051117 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:22.051086    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w22xz\""
Jul 28 21:25:22.056753 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:22.056729    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b16ceb-9sgs4"]
Jul 28 21:25:22.195626 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:22.195596    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed62079-d07a-44a9-adab-996f260562a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b16ceb-9sgs4\" (UID: \"6ed62079-d07a-44a9-adab-996f260562a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b16ceb-9sgs4"
Jul 28 21:25:22.296351 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:22.296307    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed62079-d07a-44a9-adab-996f260562a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b16ceb-9sgs4\" (UID: \"6ed62079-d07a-44a9-adab-996f260562a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b16ceb-9sgs4"
Jul 28 21:25:22.296982 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:22.296963    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed62079-d07a-44a9-adab-996f260562a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b16ceb-9sgs4\" (UID: \"6ed62079-d07a-44a9-adab-996f260562a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b16ceb-9sgs4"
Jul 28 21:25:22.362293 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:22.362258    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b16ceb-9sgs4"
Jul 28 21:25:22.497099 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:22.497074    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b16ceb-9sgs4"]
Jul 28 21:25:22.498874 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:25:22.498844    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed62079_d07a_44a9_adab_996f260562a9.slice/crio-a35b118464e0ab3796ea02a5c0cfb784b54afd2ea84125fa322fa1ac0028c946 WatchSource:0}: Error finding container a35b118464e0ab3796ea02a5c0cfb784b54afd2ea84125fa322fa1ac0028c946: Status 404 returned error can't find the container with id a35b118464e0ab3796ea02a5c0cfb784b54afd2ea84125fa322fa1ac0028c946
Jul 28 21:25:23.035344 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:23.035242    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b16ceb-9sgs4" event={"ID":"6ed62079-d07a-44a9-adab-996f260562a9","Type":"ContainerStarted","Data":"0e1b2c6211c37743f702348df63964c6c26e9a06dab211e690a861e0cd3171d9"}
Jul 28 21:25:23.035344 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:23.035303    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b16ceb-9sgs4" event={"ID":"6ed62079-d07a-44a9-adab-996f260562a9","Type":"ContainerStarted","Data":"a35b118464e0ab3796ea02a5c0cfb784b54afd2ea84125fa322fa1ac0028c946"}
Jul 28 21:25:23.049041 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:23.048978    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b16ceb-9sgs4" podStartSLOduration=1.048957055 podStartE2EDuration="1.048957055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:25:23.048143814 +0000 UTC m=+2278.272486799" watchObservedRunningTime="2026-07-28 21:25:23.048957055 +0000 UTC m=+2278.273300048"
Jul 28 21:25:24.042597 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:24.042558    2565 generic.go:356] "Generic (PLEG): container finished" podID="6ed62079-d07a-44a9-adab-996f260562a9" containerID="0e1b2c6211c37743f702348df63964c6c26e9a06dab211e690a861e0cd3171d9" exitCode=0
Jul 28 21:25:24.043010 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:24.042629    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b16ceb-9sgs4" event={"ID":"6ed62079-d07a-44a9-adab-996f260562a9","Type":"ContainerDied","Data":"0e1b2c6211c37743f702348df63964c6c26e9a06dab211e690a861e0cd3171d9"}
Jul 28 21:25:25.186561 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:25.186535    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b16ceb-9sgs4"
Jul 28 21:25:25.324888 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:25.324803    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed62079-d07a-44a9-adab-996f260562a9-maas-api-service-ca\") pod \"6ed62079-d07a-44a9-adab-996f260562a9\" (UID: \"6ed62079-d07a-44a9-adab-996f260562a9\") "
Jul 28 21:25:25.325206 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:25.325167    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed62079-d07a-44a9-adab-996f260562a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ed62079-d07a-44a9-adab-996f260562a9" (UID: "6ed62079-d07a-44a9-adab-996f260562a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:25:25.426340 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:25.426301    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed62079-d07a-44a9-adab-996f260562a9-maas-api-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:25:25.649901 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:25.649813    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:25:26.054805 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:26.054775    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b16ceb-9sgs4"
Jul 28 21:25:26.054984 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:26.054817    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b16ceb-9sgs4" event={"ID":"6ed62079-d07a-44a9-adab-996f260562a9","Type":"ContainerDied","Data":"a35b118464e0ab3796ea02a5c0cfb784b54afd2ea84125fa322fa1ac0028c946"}
Jul 28 21:25:26.054984 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:26.054863    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a35b118464e0ab3796ea02a5c0cfb784b54afd2ea84125fa322fa1ac0028c946"
Jul 28 21:25:27.244378 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:27.244349    2565 scope.go:117] "RemoveContainer" containerID="542834e467820cdfc1e2e3d9b6f57dee81109ae6d530a85c725285baae1caa0d"
Jul 28 21:25:32.503113 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:32.503078    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf"]
Jul 28 21:25:32.503720 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:32.503288    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf" podUID="cb8fc13a-6ddf-4278-befc-f023d42a5530" containerName="payload-processing" containerID="cri-o://6889950865e844a42a16b585c36a63a984042d2ed07587a98f4c81f128b50ea3" gracePeriod=30
Jul 28 21:25:32.511701 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:32.511592    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj"]
Jul 28 21:25:32.511865 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:32.511836    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj" podUID="819b169c-a7a0-40ae-b5bf-78641ac7ea91" containerName="payload-pre-processing" containerID="cri-o://a4e7d3428f5d1b320888d3fa1322a309540a16c610291e00e96016293e901451" gracePeriod=30
Jul 28 21:25:32.719219 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:32.719186    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b16ceb-9sgs4"]
Jul 28 21:25:32.727912 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:32.727882    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b16ceb-9sgs4"]
Jul 28 21:25:32.785880 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:32.785856    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf"
Jul 28 21:25:32.788797 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:32.788777    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj"
Jul 28 21:25:32.894752 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:32.894715    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8fc13a-6ddf-4278-befc-f023d42a5530-plugins-config-volume\") pod \"cb8fc13a-6ddf-4278-befc-f023d42a5530\" (UID: \"cb8fc13a-6ddf-4278-befc-f023d42a5530\") "
Jul 28 21:25:32.894932 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:32.894766    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lq9jh\" (UniqueName: \"kubernetes.io/projected/819b169c-a7a0-40ae-b5bf-78641ac7ea91-kube-api-access-lq9jh\") pod \"819b169c-a7a0-40ae-b5bf-78641ac7ea91\" (UID: \"819b169c-a7a0-40ae-b5bf-78641ac7ea91\") "
Jul 28 21:25:32.894932 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:32.894853    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/819b169c-a7a0-40ae-b5bf-78641ac7ea91-plugins-config-volume\") pod \"819b169c-a7a0-40ae-b5bf-78641ac7ea91\" (UID: \"819b169c-a7a0-40ae-b5bf-78641ac7ea91\") "
Jul 28 21:25:32.894932 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:32.894884    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdscf\" (UniqueName: \"kubernetes.io/projected/cb8fc13a-6ddf-4278-befc-f023d42a5530-kube-api-access-xdscf\") pod \"cb8fc13a-6ddf-4278-befc-f023d42a5530\" (UID: \"cb8fc13a-6ddf-4278-befc-f023d42a5530\") "
Jul 28 21:25:32.895206 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:32.895179    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8fc13a-6ddf-4278-befc-f023d42a5530-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb8fc13a-6ddf-4278-befc-f023d42a5530" (UID: "cb8fc13a-6ddf-4278-befc-f023d42a5530"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:25:32.895283 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:32.895262    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/819b169c-a7a0-40ae-b5bf-78641ac7ea91-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "819b169c-a7a0-40ae-b5bf-78641ac7ea91" (UID: "819b169c-a7a0-40ae-b5bf-78641ac7ea91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:25:32.897094 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:32.897062    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8fc13a-6ddf-4278-befc-f023d42a5530-kube-api-access-xdscf" (OuterVolumeSpecName: "kube-api-access-xdscf") pod "cb8fc13a-6ddf-4278-befc-f023d42a5530" (UID: "cb8fc13a-6ddf-4278-befc-f023d42a5530"). InnerVolumeSpecName "kube-api-access-xdscf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:25:32.897230 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:32.897131    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819b169c-a7a0-40ae-b5bf-78641ac7ea91-kube-api-access-lq9jh" (OuterVolumeSpecName: "kube-api-access-lq9jh") pod "819b169c-a7a0-40ae-b5bf-78641ac7ea91" (UID: "819b169c-a7a0-40ae-b5bf-78641ac7ea91"). InnerVolumeSpecName "kube-api-access-lq9jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:25:32.995771 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:32.995716    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/819b169c-a7a0-40ae-b5bf-78641ac7ea91-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:25:32.995771 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:32.995764    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdscf\" (UniqueName: \"kubernetes.io/projected/cb8fc13a-6ddf-4278-befc-f023d42a5530-kube-api-access-xdscf\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:25:32.995771 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:32.995773    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8fc13a-6ddf-4278-befc-f023d42a5530-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:25:32.995771 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:32.995784    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq9jh\" (UniqueName: \"kubernetes.io/projected/819b169c-a7a0-40ae-b5bf-78641ac7ea91-kube-api-access-lq9jh\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:25:33.102838 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.102732    2565 generic.go:356] "Generic (PLEG): container finished" podID="cb8fc13a-6ddf-4278-befc-f023d42a5530" containerID="6889950865e844a42a16b585c36a63a984042d2ed07587a98f4c81f128b50ea3" exitCode=0
Jul 28 21:25:33.103023 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.102901    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf"
Jul 28 21:25:33.103023 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.102963    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf" event={"ID":"cb8fc13a-6ddf-4278-befc-f023d42a5530","Type":"ContainerDied","Data":"6889950865e844a42a16b585c36a63a984042d2ed07587a98f4c81f128b50ea3"}
Jul 28 21:25:33.103138 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.103053    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf" event={"ID":"cb8fc13a-6ddf-4278-befc-f023d42a5530","Type":"ContainerDied","Data":"6e56d8eacaeff33b6297249e860df30782727b45e40da1540634e9401a0df51f"}
Jul 28 21:25:33.103138 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.103091    2565 scope.go:117] "RemoveContainer" containerID="6889950865e844a42a16b585c36a63a984042d2ed07587a98f4c81f128b50ea3"
Jul 28 21:25:33.105030 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.104879    2565 generic.go:356] "Generic (PLEG): container finished" podID="819b169c-a7a0-40ae-b5bf-78641ac7ea91" containerID="a4e7d3428f5d1b320888d3fa1322a309540a16c610291e00e96016293e901451" exitCode=0
Jul 28 21:25:33.105030 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.104963    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj" event={"ID":"819b169c-a7a0-40ae-b5bf-78641ac7ea91","Type":"ContainerDied","Data":"a4e7d3428f5d1b320888d3fa1322a309540a16c610291e00e96016293e901451"}
Jul 28 21:25:33.105030 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.105012    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj" event={"ID":"819b169c-a7a0-40ae-b5bf-78641ac7ea91","Type":"ContainerDied","Data":"cc99426010602ea8cd16ccf0d1c70b84bc32431d7c98c4234a146da163801de9"}
Jul 28 21:25:33.105210 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.105118    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj"
Jul 28 21:25:33.114593 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.114573    2565 scope.go:117] "RemoveContainer" containerID="6889950865e844a42a16b585c36a63a984042d2ed07587a98f4c81f128b50ea3"
Jul 28 21:25:33.114921 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:25:33.114903    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6889950865e844a42a16b585c36a63a984042d2ed07587a98f4c81f128b50ea3\": container with ID starting with 6889950865e844a42a16b585c36a63a984042d2ed07587a98f4c81f128b50ea3 not found: ID does not exist" containerID="6889950865e844a42a16b585c36a63a984042d2ed07587a98f4c81f128b50ea3"
Jul 28 21:25:33.114984 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.114929    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6889950865e844a42a16b585c36a63a984042d2ed07587a98f4c81f128b50ea3"} err="failed to get container status \"6889950865e844a42a16b585c36a63a984042d2ed07587a98f4c81f128b50ea3\": rpc error: code = NotFound desc = could not find container \"6889950865e844a42a16b585c36a63a984042d2ed07587a98f4c81f128b50ea3\": container with ID starting with 6889950865e844a42a16b585c36a63a984042d2ed07587a98f4c81f128b50ea3 not found: ID does not exist"
Jul 28 21:25:33.114984 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.114946    2565 scope.go:117] "RemoveContainer" containerID="a4e7d3428f5d1b320888d3fa1322a309540a16c610291e00e96016293e901451"
Jul 28 21:25:33.123545 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.123525    2565 scope.go:117] "RemoveContainer" containerID="a4e7d3428f5d1b320888d3fa1322a309540a16c610291e00e96016293e901451"
Jul 28 21:25:33.123886 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:25:33.123863    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e7d3428f5d1b320888d3fa1322a309540a16c610291e00e96016293e901451\": container with ID starting with a4e7d3428f5d1b320888d3fa1322a309540a16c610291e00e96016293e901451 not found: ID does not exist" containerID="a4e7d3428f5d1b320888d3fa1322a309540a16c610291e00e96016293e901451"
Jul 28 21:25:33.123972 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.123896    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e7d3428f5d1b320888d3fa1322a309540a16c610291e00e96016293e901451"} err="failed to get container status \"a4e7d3428f5d1b320888d3fa1322a309540a16c610291e00e96016293e901451\": rpc error: code = NotFound desc = could not find container \"a4e7d3428f5d1b320888d3fa1322a309540a16c610291e00e96016293e901451\": container with ID starting with a4e7d3428f5d1b320888d3fa1322a309540a16c610291e00e96016293e901451 not found: ID does not exist"
Jul 28 21:25:33.127740 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.127715    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf"]
Jul 28 21:25:33.130328 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.130306    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b16ceb-c6ccb9b56-kgqbf"]
Jul 28 21:25:33.141268 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.141236    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj"]
Jul 28 21:25:33.148854 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.148825    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b16ceb-744b5b5576-kwhrj"]
Jul 28 21:25:33.331791 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.331763    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ed62079-d07a-44a9-adab-996f260562a9" path="/var/lib/kubelet/pods/6ed62079-d07a-44a9-adab-996f260562a9/volumes"
Jul 28 21:25:33.332137 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.332124    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="819b169c-a7a0-40ae-b5bf-78641ac7ea91" path="/var/lib/kubelet/pods/819b169c-a7a0-40ae-b5bf-78641ac7ea91/volumes"
Jul 28 21:25:33.332449 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:33.332436    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb8fc13a-6ddf-4278-befc-f023d42a5530" path="/var/lib/kubelet/pods/cb8fc13a-6ddf-4278-befc-f023d42a5530/volumes"
Jul 28 21:25:36.919504 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:36.919462    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47" podUID="03f68325-2c63-4383-81de-f091e747d029" containerName="main" containerID="cri-o://f9086de5ab2ccedc5990bcba7cf2b4376cb4b74a7abdd7edf4b922fdf245ef1d" gracePeriod=45
Jul 28 21:25:37.130386 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.130354    2565 generic.go:356] "Generic (PLEG): container finished" podID="03f68325-2c63-4383-81de-f091e747d029" containerID="f9086de5ab2ccedc5990bcba7cf2b4376cb4b74a7abdd7edf4b922fdf245ef1d" exitCode=0
Jul 28 21:25:37.130545 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.130406    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47" event={"ID":"03f68325-2c63-4383-81de-f091e747d029","Type":"ContainerDied","Data":"f9086de5ab2ccedc5990bcba7cf2b4376cb4b74a7abdd7edf4b922fdf245ef1d"}
Jul 28 21:25:37.165516 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.165491    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:25:37.333560 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.333533    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh82k\" (UniqueName: \"kubernetes.io/projected/03f68325-2c63-4383-81de-f091e747d029-kube-api-access-hh82k\") pod \"03f68325-2c63-4383-81de-f091e747d029\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") "
Jul 28 21:25:37.333756 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.333574    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03f68325-2c63-4383-81de-f091e747d029-tls-certs\") pod \"03f68325-2c63-4383-81de-f091e747d029\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") "
Jul 28 21:25:37.333756 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.333600    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-dshm\") pod \"03f68325-2c63-4383-81de-f091e747d029\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") "
Jul 28 21:25:37.333756 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.333675    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-tmp-dir\") pod \"03f68325-2c63-4383-81de-f091e747d029\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") "
Jul 28 21:25:37.333756 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.333727    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-model-cache\") pod \"03f68325-2c63-4383-81de-f091e747d029\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") "
Jul 28 21:25:37.333979 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.333775    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-home\") pod \"03f68325-2c63-4383-81de-f091e747d029\" (UID: \"03f68325-2c63-4383-81de-f091e747d029\") "
Jul 28 21:25:37.333979 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.333913    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "03f68325-2c63-4383-81de-f091e747d029" (UID: "03f68325-2c63-4383-81de-f091e747d029"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:25:37.334070 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.333997    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-model-cache" (OuterVolumeSpecName: "model-cache") pod "03f68325-2c63-4383-81de-f091e747d029" (UID: "03f68325-2c63-4383-81de-f091e747d029"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:25:37.334117 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.334088    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-home" (OuterVolumeSpecName: "home") pod "03f68325-2c63-4383-81de-f091e747d029" (UID: "03f68325-2c63-4383-81de-f091e747d029"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:25:37.334153 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.334133    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-tmp-dir\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:25:37.334153 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.334147    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-model-cache\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:25:37.335901 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.335874    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-dshm" (OuterVolumeSpecName: "dshm") pod "03f68325-2c63-4383-81de-f091e747d029" (UID: "03f68325-2c63-4383-81de-f091e747d029"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:25:37.336141 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.336123    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03f68325-2c63-4383-81de-f091e747d029-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03f68325-2c63-4383-81de-f091e747d029" (UID: "03f68325-2c63-4383-81de-f091e747d029"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:25:37.336208 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.336139    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f68325-2c63-4383-81de-f091e747d029-kube-api-access-hh82k" (OuterVolumeSpecName: "kube-api-access-hh82k") pod "03f68325-2c63-4383-81de-f091e747d029" (UID: "03f68325-2c63-4383-81de-f091e747d029"). InnerVolumeSpecName "kube-api-access-hh82k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:25:37.434967 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.434935    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-home\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:25:37.434967 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.434962    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh82k\" (UniqueName: \"kubernetes.io/projected/03f68325-2c63-4383-81de-f091e747d029-kube-api-access-hh82k\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:25:37.434967 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.434973    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03f68325-2c63-4383-81de-f091e747d029-tls-certs\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:25:37.435205 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:37.434981    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03f68325-2c63-4383-81de-f091e747d029-dshm\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:25:38.053688 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:38.053656    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:25:38.136651 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:38.136605    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"
Jul 28 21:25:38.136799 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:38.136603    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47" event={"ID":"03f68325-2c63-4383-81de-f091e747d029","Type":"ContainerDied","Data":"aeae6ad87d25a5ec670f5fb1f7b7d9c605454d246ea2a3494bc7aaf602b23f19"}
Jul 28 21:25:38.136799 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:38.136750    2565 scope.go:117] "RemoveContainer" containerID="f9086de5ab2ccedc5990bcba7cf2b4376cb4b74a7abdd7edf4b922fdf245ef1d"
Jul 28 21:25:38.160667 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:38.160612    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"]
Jul 28 21:25:38.164052 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:38.164028    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-75f962/test-model-75f962-kserve-699569c979-mbm47"]
Jul 28 21:25:39.331996 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.331962    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03f68325-2c63-4383-81de-f091e747d029" path="/var/lib/kubelet/pods/03f68325-2c63-4383-81de-f091e747d029/volumes"
Jul 28 21:25:39.504973 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.504939    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75f962-tp6pq"]
Jul 28 21:25:39.505282 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.505270    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb8fc13a-6ddf-4278-befc-f023d42a5530" containerName="payload-processing"
Jul 28 21:25:39.505333 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.505284    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb8fc13a-6ddf-4278-befc-f023d42a5530" containerName="payload-processing"
Jul 28 21:25:39.505333 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.505293    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f68325-2c63-4383-81de-f091e747d029" containerName="main"
Jul 28 21:25:39.505333 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.505299    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f68325-2c63-4383-81de-f091e747d029" containerName="main"
Jul 28 21:25:39.505333 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.505312    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="819b169c-a7a0-40ae-b5bf-78641ac7ea91" containerName="payload-pre-processing"
Jul 28 21:25:39.505333 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.505318    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="819b169c-a7a0-40ae-b5bf-78641ac7ea91" containerName="payload-pre-processing"
Jul 28 21:25:39.505333 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.505333    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ed62079-d07a-44a9-adab-996f260562a9" containerName="revoke-keys"
Jul 28 21:25:39.505513 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.505339    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ed62079-d07a-44a9-adab-996f260562a9" containerName="revoke-keys"
Jul 28 21:25:39.505513 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.505405    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="03f68325-2c63-4383-81de-f091e747d029" containerName="main"
Jul 28 21:25:39.505513 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.505415    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb8fc13a-6ddf-4278-befc-f023d42a5530" containerName="payload-processing"
Jul 28 21:25:39.505513 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.505421    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="819b169c-a7a0-40ae-b5bf-78641ac7ea91" containerName="payload-pre-processing"
Jul 28 21:25:39.505513 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.505426    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ed62079-d07a-44a9-adab-996f260562a9" containerName="revoke-keys"
Jul 28 21:25:39.509627 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.509605    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75f962-tp6pq"
Jul 28 21:25:39.511599 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.511579    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 21:25:39.511702 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.511620    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w22xz\""
Jul 28 21:25:39.516007 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.515983    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75f962-tp6pq"]
Jul 28 21:25:39.657198 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.657099    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a62f733-2dcc-4563-b9d8-e2dd0dd029ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-75f962-tp6pq\" (UID: \"2a62f733-2dcc-4563-b9d8-e2dd0dd029ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75f962-tp6pq"
Jul 28 21:25:39.757949 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.757913    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a62f733-2dcc-4563-b9d8-e2dd0dd029ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-75f962-tp6pq\" (UID: \"2a62f733-2dcc-4563-b9d8-e2dd0dd029ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75f962-tp6pq"
Jul 28 21:25:39.758526 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.758504    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a62f733-2dcc-4563-b9d8-e2dd0dd029ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-75f962-tp6pq\" (UID: \"2a62f733-2dcc-4563-b9d8-e2dd0dd029ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75f962-tp6pq"
Jul 28 21:25:39.821983 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.821935    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75f962-tp6pq"
Jul 28 21:25:39.952904 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:39.952877    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75f962-tp6pq"]
Jul 28 21:25:39.954713 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:25:39.954687    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a62f733_2dcc_4563_b9d8_e2dd0dd029ae.slice/crio-ee404ed5cbfbbe8f6d75fa3b7bfe4e7d0acbf9db46a304b11cd5b2b8f0fc1c8a WatchSource:0}: Error finding container ee404ed5cbfbbe8f6d75fa3b7bfe4e7d0acbf9db46a304b11cd5b2b8f0fc1c8a: Status 404 returned error can't find the container with id ee404ed5cbfbbe8f6d75fa3b7bfe4e7d0acbf9db46a304b11cd5b2b8f0fc1c8a
Jul 28 21:25:40.149921 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:40.149886    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75f962-tp6pq" event={"ID":"2a62f733-2dcc-4563-b9d8-e2dd0dd029ae","Type":"ContainerStarted","Data":"ee404ed5cbfbbe8f6d75fa3b7bfe4e7d0acbf9db46a304b11cd5b2b8f0fc1c8a"}
Jul 28 21:25:41.157281 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:41.157247    2565 generic.go:356] "Generic (PLEG): container finished" podID="2a62f733-2dcc-4563-b9d8-e2dd0dd029ae" containerID="acc51019bce1d89b1f2967acc6f8487a38a4193bbd626c137cc2f44fd70ded49" exitCode=0
Jul 28 21:25:41.157714 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:41.157311    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75f962-tp6pq" event={"ID":"2a62f733-2dcc-4563-b9d8-e2dd0dd029ae","Type":"ContainerDied","Data":"acc51019bce1d89b1f2967acc6f8487a38a4193bbd626c137cc2f44fd70ded49"}
Jul 28 21:25:42.301452 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:42.301423    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75f962-tp6pq"
Jul 28 21:25:42.481533 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:42.481498    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a62f733-2dcc-4563-b9d8-e2dd0dd029ae-maas-api-service-ca\") pod \"2a62f733-2dcc-4563-b9d8-e2dd0dd029ae\" (UID: \"2a62f733-2dcc-4563-b9d8-e2dd0dd029ae\") "
Jul 28 21:25:42.481886 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:42.481863    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a62f733-2dcc-4563-b9d8-e2dd0dd029ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2a62f733-2dcc-4563-b9d8-e2dd0dd029ae" (UID: "2a62f733-2dcc-4563-b9d8-e2dd0dd029ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:25:42.582674 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:42.582614    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a62f733-2dcc-4563-b9d8-e2dd0dd029ae-maas-api-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:25:43.169204 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:43.169172    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75f962-tp6pq"
Jul 28 21:25:43.169376 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:43.169214    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75f962-tp6pq" event={"ID":"2a62f733-2dcc-4563-b9d8-e2dd0dd029ae","Type":"ContainerDied","Data":"ee404ed5cbfbbe8f6d75fa3b7bfe4e7d0acbf9db46a304b11cd5b2b8f0fc1c8a"}
Jul 28 21:25:43.169376 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:43.169268    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee404ed5cbfbbe8f6d75fa3b7bfe4e7d0acbf9db46a304b11cd5b2b8f0fc1c8a"
Jul 28 21:25:50.648954 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:50.648912    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr"]
Jul 28 21:25:50.649433 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:50.649138    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr" podUID="3015f78e-a332-4c4c-b538-61ae496d68b1" containerName="payload-processing" containerID="cri-o://cce9e48882a9091514b1eb670015751cf96f05abfc98b8fc8dcd75f467dea44e" gracePeriod=30
Jul 28 21:25:50.655462 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:50.655430    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7"]
Jul 28 21:25:50.655720 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:50.655682    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7" podUID="f2b8fb7f-a7ba-4c72-974b-92403f5b0234" containerName="payload-pre-processing" containerID="cri-o://cee651562ec35faed0f6ed18ea47c40e2bc925a0ea2549c5a84bc3858af8356a" gracePeriod=30
Jul 28 21:25:50.855922 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:50.855846    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75f962-tp6pq"]
Jul 28 21:25:50.858194 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:50.858169    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75f962-tp6pq"]
Jul 28 21:25:50.927143 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:50.927122    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr"
Jul 28 21:25:50.929759 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:50.929732    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7"
Jul 28 21:25:50.954615 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:50.954589    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5g5w\" (UniqueName: \"kubernetes.io/projected/3015f78e-a332-4c4c-b538-61ae496d68b1-kube-api-access-c5g5w\") pod \"3015f78e-a332-4c4c-b538-61ae496d68b1\" (UID: \"3015f78e-a332-4c4c-b538-61ae496d68b1\") "
Jul 28 21:25:50.954773 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:50.954629    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3015f78e-a332-4c4c-b538-61ae496d68b1-plugins-config-volume\") pod \"3015f78e-a332-4c4c-b538-61ae496d68b1\" (UID: \"3015f78e-a332-4c4c-b538-61ae496d68b1\") "
Jul 28 21:25:50.954773 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:50.954679    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b8fb7f-a7ba-4c72-974b-92403f5b0234-plugins-config-volume\") pod \"f2b8fb7f-a7ba-4c72-974b-92403f5b0234\" (UID: \"f2b8fb7f-a7ba-4c72-974b-92403f5b0234\") "
Jul 28 21:25:50.954773 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:50.954695    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-599l9\" (UniqueName: \"kubernetes.io/projected/f2b8fb7f-a7ba-4c72-974b-92403f5b0234-kube-api-access-599l9\") pod \"f2b8fb7f-a7ba-4c72-974b-92403f5b0234\" (UID: \"f2b8fb7f-a7ba-4c72-974b-92403f5b0234\") "
Jul 28 21:25:50.955129 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:50.955092    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3015f78e-a332-4c4c-b538-61ae496d68b1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3015f78e-a332-4c4c-b538-61ae496d68b1" (UID: "3015f78e-a332-4c4c-b538-61ae496d68b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:25:50.955129 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:50.955093    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b8fb7f-a7ba-4c72-974b-92403f5b0234-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2b8fb7f-a7ba-4c72-974b-92403f5b0234" (UID: "f2b8fb7f-a7ba-4c72-974b-92403f5b0234"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:25:50.956942 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:50.956914    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b8fb7f-a7ba-4c72-974b-92403f5b0234-kube-api-access-599l9" (OuterVolumeSpecName: "kube-api-access-599l9") pod "f2b8fb7f-a7ba-4c72-974b-92403f5b0234" (UID: "f2b8fb7f-a7ba-4c72-974b-92403f5b0234"). InnerVolumeSpecName "kube-api-access-599l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:25:50.957063 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:50.957001    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3015f78e-a332-4c4c-b538-61ae496d68b1-kube-api-access-c5g5w" (OuterVolumeSpecName: "kube-api-access-c5g5w") pod "3015f78e-a332-4c4c-b538-61ae496d68b1" (UID: "3015f78e-a332-4c4c-b538-61ae496d68b1"). InnerVolumeSpecName "kube-api-access-c5g5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:25:51.055862 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.055813    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5g5w\" (UniqueName: \"kubernetes.io/projected/3015f78e-a332-4c4c-b538-61ae496d68b1-kube-api-access-c5g5w\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:25:51.055862 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.055854    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3015f78e-a332-4c4c-b538-61ae496d68b1-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:25:51.055862 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.055863    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2b8fb7f-a7ba-4c72-974b-92403f5b0234-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:25:51.055862 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.055872    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-599l9\" (UniqueName: \"kubernetes.io/projected/f2b8fb7f-a7ba-4c72-974b-92403f5b0234-kube-api-access-599l9\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:25:51.216397 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.215705    2565 generic.go:356] "Generic (PLEG): container finished" podID="3015f78e-a332-4c4c-b538-61ae496d68b1" containerID="cce9e48882a9091514b1eb670015751cf96f05abfc98b8fc8dcd75f467dea44e" exitCode=0
Jul 28 21:25:51.216397 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.215920    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr" event={"ID":"3015f78e-a332-4c4c-b538-61ae496d68b1","Type":"ContainerDied","Data":"cce9e48882a9091514b1eb670015751cf96f05abfc98b8fc8dcd75f467dea44e"}
Jul 28 21:25:51.216397 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.215961    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr"
Jul 28 21:25:51.216397 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.216088    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr" event={"ID":"3015f78e-a332-4c4c-b538-61ae496d68b1","Type":"ContainerDied","Data":"c02002e23b41344ccc845483af36f8570be974609c6900e46f2e29d90c61c738"}
Jul 28 21:25:51.216397 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.216138    2565 scope.go:117] "RemoveContainer" containerID="cce9e48882a9091514b1eb670015751cf96f05abfc98b8fc8dcd75f467dea44e"
Jul 28 21:25:51.223744 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.223719    2565 generic.go:356] "Generic (PLEG): container finished" podID="f2b8fb7f-a7ba-4c72-974b-92403f5b0234" containerID="cee651562ec35faed0f6ed18ea47c40e2bc925a0ea2549c5a84bc3858af8356a" exitCode=0
Jul 28 21:25:51.223880 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.223778    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7"
Jul 28 21:25:51.223880 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.223815    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7" event={"ID":"f2b8fb7f-a7ba-4c72-974b-92403f5b0234","Type":"ContainerDied","Data":"cee651562ec35faed0f6ed18ea47c40e2bc925a0ea2549c5a84bc3858af8356a"}
Jul 28 21:25:51.223880 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.223868    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7" event={"ID":"f2b8fb7f-a7ba-4c72-974b-92403f5b0234","Type":"ContainerDied","Data":"61b881bf9f4bc988d5633249920cba3c3569b5f0aa347cf0887900d9c90c3d0c"}
Jul 28 21:25:51.232441 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.232284    2565 scope.go:117] "RemoveContainer" containerID="cce9e48882a9091514b1eb670015751cf96f05abfc98b8fc8dcd75f467dea44e"
Jul 28 21:25:51.232977 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:25:51.232883    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce9e48882a9091514b1eb670015751cf96f05abfc98b8fc8dcd75f467dea44e\": container with ID starting with cce9e48882a9091514b1eb670015751cf96f05abfc98b8fc8dcd75f467dea44e not found: ID does not exist" containerID="cce9e48882a9091514b1eb670015751cf96f05abfc98b8fc8dcd75f467dea44e"
Jul 28 21:25:51.232977 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.232916    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce9e48882a9091514b1eb670015751cf96f05abfc98b8fc8dcd75f467dea44e"} err="failed to get container status \"cce9e48882a9091514b1eb670015751cf96f05abfc98b8fc8dcd75f467dea44e\": rpc error: code = NotFound desc = could not find container \"cce9e48882a9091514b1eb670015751cf96f05abfc98b8fc8dcd75f467dea44e\": container with ID starting with cce9e48882a9091514b1eb670015751cf96f05abfc98b8fc8dcd75f467dea44e not found: ID does not exist"
Jul 28 21:25:51.232977 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.232940    2565 scope.go:117] "RemoveContainer" containerID="cee651562ec35faed0f6ed18ea47c40e2bc925a0ea2549c5a84bc3858af8356a"
Jul 28 21:25:51.241461 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.241435    2565 scope.go:117] "RemoveContainer" containerID="cee651562ec35faed0f6ed18ea47c40e2bc925a0ea2549c5a84bc3858af8356a"
Jul 28 21:25:51.241713 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:25:51.241694    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee651562ec35faed0f6ed18ea47c40e2bc925a0ea2549c5a84bc3858af8356a\": container with ID starting with cee651562ec35faed0f6ed18ea47c40e2bc925a0ea2549c5a84bc3858af8356a not found: ID does not exist" containerID="cee651562ec35faed0f6ed18ea47c40e2bc925a0ea2549c5a84bc3858af8356a"
Jul 28 21:25:51.241773 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.241720    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee651562ec35faed0f6ed18ea47c40e2bc925a0ea2549c5a84bc3858af8356a"} err="failed to get container status \"cee651562ec35faed0f6ed18ea47c40e2bc925a0ea2549c5a84bc3858af8356a\": rpc error: code = NotFound desc = could not find container \"cee651562ec35faed0f6ed18ea47c40e2bc925a0ea2549c5a84bc3858af8356a\": container with ID starting with cee651562ec35faed0f6ed18ea47c40e2bc925a0ea2549c5a84bc3858af8356a not found: ID does not exist"
Jul 28 21:25:51.241814 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.241794    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr"]
Jul 28 21:25:51.247104 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.247086    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-75f962-7f545fd846-rd7sr"]
Jul 28 21:25:51.255019 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.254999    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7"]
Jul 28 21:25:51.260897 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.260876    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-75f962-59b87d9d44-nv8q7"]
Jul 28 21:25:51.331651 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.331605    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a62f733-2dcc-4563-b9d8-e2dd0dd029ae" path="/var/lib/kubelet/pods/2a62f733-2dcc-4563-b9d8-e2dd0dd029ae/volumes"
Jul 28 21:25:51.332024 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.332010    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3015f78e-a332-4c4c-b538-61ae496d68b1" path="/var/lib/kubelet/pods/3015f78e-a332-4c4c-b538-61ae496d68b1/volumes"
Jul 28 21:25:51.332343 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:51.332329    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2b8fb7f-a7ba-4c72-974b-92403f5b0234" path="/var/lib/kubelet/pods/f2b8fb7f-a7ba-4c72-974b-92403f5b0234/volumes"
Jul 28 21:25:53.356507 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:25:53.356468    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:26:02.044667 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.044616    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9"]
Jul 28 21:26:02.045062 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.044943    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a62f733-2dcc-4563-b9d8-e2dd0dd029ae" containerName="revoke-keys"
Jul 28 21:26:02.045062 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.044955    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a62f733-2dcc-4563-b9d8-e2dd0dd029ae" containerName="revoke-keys"
Jul 28 21:26:02.045062 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.044978    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b8fb7f-a7ba-4c72-974b-92403f5b0234" containerName="payload-pre-processing"
Jul 28 21:26:02.045062 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.044984    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b8fb7f-a7ba-4c72-974b-92403f5b0234" containerName="payload-pre-processing"
Jul 28 21:26:02.045062 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.044992    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3015f78e-a332-4c4c-b538-61ae496d68b1" containerName="payload-processing"
Jul 28 21:26:02.045062 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.044997    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3015f78e-a332-4c4c-b538-61ae496d68b1" containerName="payload-processing"
Jul 28 21:26:02.045062 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.045047    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3015f78e-a332-4c4c-b538-61ae496d68b1" containerName="payload-processing"
Jul 28 21:26:02.045062 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.045057    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2b8fb7f-a7ba-4c72-974b-92403f5b0234" containerName="payload-pre-processing"
Jul 28 21:26:02.045062 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.045065    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a62f733-2dcc-4563-b9d8-e2dd0dd029ae" containerName="revoke-keys"
Jul 28 21:26:02.048083 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.048067    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9"
Jul 28 21:26:02.050082 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.050055    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3c4e74\""
Jul 28 21:26:02.050193 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.050132    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3c4e74-dockercfg-rgj97\""
Jul 28 21:26:02.058709 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.058683    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9"]
Jul 28 21:26:02.084278 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.084241    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k"]
Jul 28 21:26:02.087553 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.087535    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k"
Jul 28 21:26:02.100268 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.100242    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k"]
Jul 28 21:26:02.154012 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.153981    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvbgl\" (UniqueName: \"kubernetes.io/projected/b1844b76-2ef1-470b-baa9-4a62d25f782f-kube-api-access-qvbgl\") pod \"payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9\" (UID: \"b1844b76-2ef1-470b-baa9-4a62d25f782f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9"
Jul 28 21:26:02.154181 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.154031    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1844b76-2ef1-470b-baa9-4a62d25f782f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9\" (UID: \"b1844b76-2ef1-470b-baa9-4a62d25f782f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9"
Jul 28 21:26:02.154181 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.154060    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/176b0935-7d01-4063-b179-725fd5995f91-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k\" (UID: \"176b0935-7d01-4063-b179-725fd5995f91\") " pod="openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k"
Jul 28 21:26:02.154181 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.154118    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wfql\" (UniqueName: \"kubernetes.io/projected/176b0935-7d01-4063-b179-725fd5995f91-kube-api-access-7wfql\") pod \"payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k\" (UID: \"176b0935-7d01-4063-b179-725fd5995f91\") " pod="openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k"
Jul 28 21:26:02.254859 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.254817    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wfql\" (UniqueName: \"kubernetes.io/projected/176b0935-7d01-4063-b179-725fd5995f91-kube-api-access-7wfql\") pod \"payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k\" (UID: \"176b0935-7d01-4063-b179-725fd5995f91\") " pod="openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k"
Jul 28 21:26:02.255070 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.254868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvbgl\" (UniqueName: \"kubernetes.io/projected/b1844b76-2ef1-470b-baa9-4a62d25f782f-kube-api-access-qvbgl\") pod \"payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9\" (UID: \"b1844b76-2ef1-470b-baa9-4a62d25f782f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9"
Jul 28 21:26:02.255070 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.254907    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1844b76-2ef1-470b-baa9-4a62d25f782f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9\" (UID: \"b1844b76-2ef1-470b-baa9-4a62d25f782f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9"
Jul 28 21:26:02.255070 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.254923    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/176b0935-7d01-4063-b179-725fd5995f91-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k\" (UID: \"176b0935-7d01-4063-b179-725fd5995f91\") " pod="openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k"
Jul 28 21:26:02.255882 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.255855    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1844b76-2ef1-470b-baa9-4a62d25f782f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9\" (UID: \"b1844b76-2ef1-470b-baa9-4a62d25f782f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9"
Jul 28 21:26:02.255882 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.255875    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/176b0935-7d01-4063-b179-725fd5995f91-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k\" (UID: \"176b0935-7d01-4063-b179-725fd5995f91\") " pod="openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k"
Jul 28 21:26:02.264849 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.264815    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvbgl\" (UniqueName: \"kubernetes.io/projected/b1844b76-2ef1-470b-baa9-4a62d25f782f-kube-api-access-qvbgl\") pod \"payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9\" (UID: \"b1844b76-2ef1-470b-baa9-4a62d25f782f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9"
Jul 28 21:26:02.265052 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.265030    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wfql\" (UniqueName: \"kubernetes.io/projected/176b0935-7d01-4063-b179-725fd5995f91-kube-api-access-7wfql\") pod \"payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k\" (UID: \"176b0935-7d01-4063-b179-725fd5995f91\") " pod="openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k"
Jul 28 21:26:02.359112 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.359023    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9"
Jul 28 21:26:02.397840 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.397804    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k"
Jul 28 21:26:02.507452 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.507419    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9"]
Jul 28 21:26:02.507992 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:26:02.507950    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1844b76_2ef1_470b_baa9_4a62d25f782f.slice/crio-19e56745f04cdd0a7a3bf6736cf538e765e6d6cf7399ed0a88272d184f7bc19d WatchSource:0}: Error finding container 19e56745f04cdd0a7a3bf6736cf538e765e6d6cf7399ed0a88272d184f7bc19d: Status 404 returned error can't find the container with id 19e56745f04cdd0a7a3bf6736cf538e765e6d6cf7399ed0a88272d184f7bc19d
Jul 28 21:26:02.548030 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:02.548009    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k"]
Jul 28 21:26:02.550509 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:26:02.550483    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176b0935_7d01_4063_b179_725fd5995f91.slice/crio-a691869040ff7c718bc9834da811d389fffa1835865ef87f2d7c99a7a4f27c76 WatchSource:0}: Error finding container a691869040ff7c718bc9834da811d389fffa1835865ef87f2d7c99a7a4f27c76: Status 404 returned error can't find the container with id a691869040ff7c718bc9834da811d389fffa1835865ef87f2d7c99a7a4f27c76
Jul 28 21:26:03.290027 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:03.289984    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9" event={"ID":"b1844b76-2ef1-470b-baa9-4a62d25f782f","Type":"ContainerStarted","Data":"d6b3e9590321881e3d124d7757723d0e793741809c62d47a0738043b58aa7c29"}
Jul 28 21:26:03.290595 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:03.290056    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9" event={"ID":"b1844b76-2ef1-470b-baa9-4a62d25f782f","Type":"ContainerStarted","Data":"19e56745f04cdd0a7a3bf6736cf538e765e6d6cf7399ed0a88272d184f7bc19d"}
Jul 28 21:26:03.290595 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:03.290233    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9"
Jul 28 21:26:03.291683 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:03.291655    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k" event={"ID":"176b0935-7d01-4063-b179-725fd5995f91","Type":"ContainerStarted","Data":"4e27debf4d331ab80fb4e2f312a25e0478036e43c382b9fffd2e0950210daa1c"}
Jul 28 21:26:03.291823 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:03.291696    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k" event={"ID":"176b0935-7d01-4063-b179-725fd5995f91","Type":"ContainerStarted","Data":"a691869040ff7c718bc9834da811d389fffa1835865ef87f2d7c99a7a4f27c76"}
Jul 28 21:26:03.291823 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:03.291740    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k"
Jul 28 21:26:03.312117 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:03.312072    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9" podStartSLOduration=1.312058551 podStartE2EDuration="1.312058551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:26:03.310227311 +0000 UTC m=+2318.534570296" watchObservedRunningTime="2026-07-28 21:26:03.312058551 +0000 UTC m=+2318.536401583"
Jul 28 21:26:03.328682 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:03.328626    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k" podStartSLOduration=1.328611351 podStartE2EDuration="1.328611351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:26:03.327828845 +0000 UTC m=+2318.552171832" watchObservedRunningTime="2026-07-28 21:26:03.328611351 +0000 UTC m=+2318.552954336"
Jul 28 21:26:07.765365 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:07.765326    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:26:14.298739 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:14.298706    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k"
Jul 28 21:26:14.299136 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:14.298781    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9"
Jul 28 21:26:30.377355 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:30.377317    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"]
Jul 28 21:26:30.382708 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:30.382679    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"
Jul 28 21:26:30.384848 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:30.384822    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-65cc6c-serving-cert\""
Jul 28 21:26:30.384982 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:30.384914    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 21:26:30.385050 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:30.384990    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2thqx\""
Jul 28 21:26:30.385170 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:30.385155    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 21:26:30.394890 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:30.394864    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"]
Jul 28 21:26:30.501647 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:30.501597    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4e9546-5762-426f-b651-d9cd5c9a7c20-openshift-service-ca\") pod \"maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4\" (UID: \"8f4e9546-5762-426f-b651-d9cd5c9a7c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"
Jul 28 21:26:30.501829 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:30.501697    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlkst\" (UniqueName: \"kubernetes.io/projected/8f4e9546-5762-426f-b651-d9cd5c9a7c20-kube-api-access-wlkst\") pod \"maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4\" (UID: \"8f4e9546-5762-426f-b651-d9cd5c9a7c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"
Jul 28 21:26:30.501829 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:30.501739    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4e9546-5762-426f-b651-d9cd5c9a7c20-maas-api-tls\") pod \"maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4\" (UID: \"8f4e9546-5762-426f-b651-d9cd5c9a7c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"
Jul 28 21:26:30.602159 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:30.602119    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4e9546-5762-426f-b651-d9cd5c9a7c20-openshift-service-ca\") pod \"maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4\" (UID: \"8f4e9546-5762-426f-b651-d9cd5c9a7c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"
Jul 28 21:26:30.602323 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:30.602200    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlkst\" (UniqueName: \"kubernetes.io/projected/8f4e9546-5762-426f-b651-d9cd5c9a7c20-kube-api-access-wlkst\") pod \"maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4\" (UID: \"8f4e9546-5762-426f-b651-d9cd5c9a7c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"
Jul 28 21:26:30.602323 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:30.602244    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4e9546-5762-426f-b651-d9cd5c9a7c20-maas-api-tls\") pod \"maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4\" (UID: \"8f4e9546-5762-426f-b651-d9cd5c9a7c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"
Jul 28 21:26:30.602403 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:26:30.602367    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-serving-cert: secret "maas-api-e2e-api-b-65cc6c-serving-cert" not found
Jul 28 21:26:30.602459 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:26:30.602447    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4e9546-5762-426f-b651-d9cd5c9a7c20-maas-api-tls podName:8f4e9546-5762-426f-b651-d9cd5c9a7c20 nodeName:}" failed. No retries permitted until 2026-07-28 21:26:31.102425315 +0000 UTC m=+2346.326768280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f4e9546-5762-426f-b651-d9cd5c9a7c20-maas-api-tls") pod "maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4" (UID: "8f4e9546-5762-426f-b651-d9cd5c9a7c20") : secret "maas-api-e2e-api-b-65cc6c-serving-cert" not found
Jul 28 21:26:30.602765 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:30.602746    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4e9546-5762-426f-b651-d9cd5c9a7c20-openshift-service-ca\") pod \"maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4\" (UID: \"8f4e9546-5762-426f-b651-d9cd5c9a7c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"
Jul 28 21:26:30.616716 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:30.616683    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlkst\" (UniqueName: \"kubernetes.io/projected/8f4e9546-5762-426f-b651-d9cd5c9a7c20-kube-api-access-wlkst\") pod \"maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4\" (UID: \"8f4e9546-5762-426f-b651-d9cd5c9a7c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"
Jul 28 21:26:31.106797 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.106764    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4e9546-5762-426f-b651-d9cd5c9a7c20-maas-api-tls\") pod \"maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4\" (UID: \"8f4e9546-5762-426f-b651-d9cd5c9a7c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"
Jul 28 21:26:31.109450 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.109416    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4e9546-5762-426f-b651-d9cd5c9a7c20-maas-api-tls\") pod \"maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4\" (UID: \"8f4e9546-5762-426f-b651-d9cd5c9a7c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"
Jul 28 21:26:31.205401 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.205362    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v"]
Jul 28 21:26:31.208799 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.208777    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v"
Jul 28 21:26:31.210839 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.210818    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-65cc6c\""
Jul 28 21:26:31.210935 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.210914    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-65cc6c-dockercfg-mn9jd\""
Jul 28 21:26:31.214928 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.214906    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj"]
Jul 28 21:26:31.218363 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.218346    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj"
Jul 28 21:26:31.219141 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.219121    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v"]
Jul 28 21:26:31.230800 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.230777    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj"]
Jul 28 21:26:31.294730 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.294696    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"
Jul 28 21:26:31.308021 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.307992    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44811dcd-c92f-47cb-ba20-84af02416fc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v\" (UID: \"44811dcd-c92f-47cb-ba20-84af02416fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v"
Jul 28 21:26:31.308158 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.308061    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdgl5\" (UniqueName: \"kubernetes.io/projected/44811dcd-c92f-47cb-ba20-84af02416fc5-kube-api-access-qdgl5\") pod \"payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v\" (UID: \"44811dcd-c92f-47cb-ba20-84af02416fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v"
Jul 28 21:26:31.409233 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.409197    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a402b0c-5d3a-4081-a235-bb1eb0f5893c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj\" (UID: \"0a402b0c-5d3a-4081-a235-bb1eb0f5893c\") " pod="openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj"
Jul 28 21:26:31.409706 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.409242    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h98rj\" (UniqueName: \"kubernetes.io/projected/0a402b0c-5d3a-4081-a235-bb1eb0f5893c-kube-api-access-h98rj\") pod \"payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj\" (UID: \"0a402b0c-5d3a-4081-a235-bb1eb0f5893c\") " pod="openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj"
Jul 28 21:26:31.409706 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.409264    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44811dcd-c92f-47cb-ba20-84af02416fc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v\" (UID: \"44811dcd-c92f-47cb-ba20-84af02416fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v"
Jul 28 21:26:31.409706 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.409341    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdgl5\" (UniqueName: \"kubernetes.io/projected/44811dcd-c92f-47cb-ba20-84af02416fc5-kube-api-access-qdgl5\") pod \"payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v\" (UID: \"44811dcd-c92f-47cb-ba20-84af02416fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v"
Jul 28 21:26:31.410041 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.410021    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44811dcd-c92f-47cb-ba20-84af02416fc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v\" (UID: \"44811dcd-c92f-47cb-ba20-84af02416fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v"
Jul 28 21:26:31.418969 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.418943    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdgl5\" (UniqueName: \"kubernetes.io/projected/44811dcd-c92f-47cb-ba20-84af02416fc5-kube-api-access-qdgl5\") pod \"payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v\" (UID: \"44811dcd-c92f-47cb-ba20-84af02416fc5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v"
Jul 28 21:26:31.433232 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.433209    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"]
Jul 28 21:26:31.435321 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:26:31.435295    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4e9546_5762_426f_b651_d9cd5c9a7c20.slice/crio-2f57602e833dc890963ccabfba57e427d1216c6428985813aa2a66d467e4f373 WatchSource:0}: Error finding container 2f57602e833dc890963ccabfba57e427d1216c6428985813aa2a66d467e4f373: Status 404 returned error can't find the container with id 2f57602e833dc890963ccabfba57e427d1216c6428985813aa2a66d467e4f373
Jul 28 21:26:31.446547 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.446517    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4" event={"ID":"8f4e9546-5762-426f-b651-d9cd5c9a7c20","Type":"ContainerStarted","Data":"2f57602e833dc890963ccabfba57e427d1216c6428985813aa2a66d467e4f373"}
Jul 28 21:26:31.510574 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.510522    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a402b0c-5d3a-4081-a235-bb1eb0f5893c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj\" (UID: \"0a402b0c-5d3a-4081-a235-bb1eb0f5893c\") " pod="openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj"
Jul 28 21:26:31.510574 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.510585    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h98rj\" (UniqueName: \"kubernetes.io/projected/0a402b0c-5d3a-4081-a235-bb1eb0f5893c-kube-api-access-h98rj\") pod \"payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj\" (UID: \"0a402b0c-5d3a-4081-a235-bb1eb0f5893c\") " pod="openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj"
Jul 28 21:26:31.511299 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.511272    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a402b0c-5d3a-4081-a235-bb1eb0f5893c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj\" (UID: \"0a402b0c-5d3a-4081-a235-bb1eb0f5893c\") " pod="openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj"
Jul 28 21:26:31.518461 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.518437    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h98rj\" (UniqueName: \"kubernetes.io/projected/0a402b0c-5d3a-4081-a235-bb1eb0f5893c-kube-api-access-h98rj\") pod \"payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj\" (UID: \"0a402b0c-5d3a-4081-a235-bb1eb0f5893c\") " pod="openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj"
Jul 28 21:26:31.520252 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.520235    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v"
Jul 28 21:26:31.528533 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.528516    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj"
Jul 28 21:26:31.673069 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.673042    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v"]
Jul 28 21:26:31.675203 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:26:31.675171    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44811dcd_c92f_47cb_ba20_84af02416fc5.slice/crio-ded6bb56935118d612bd694a5bede4c595b97c39da128a14b70d3e4fe3412519 WatchSource:0}: Error finding container ded6bb56935118d612bd694a5bede4c595b97c39da128a14b70d3e4fe3412519: Status 404 returned error can't find the container with id ded6bb56935118d612bd694a5bede4c595b97c39da128a14b70d3e4fe3412519
Jul 28 21:26:31.693602 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:31.693578    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj"]
Jul 28 21:26:31.696287 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:26:31.696257    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a402b0c_5d3a_4081_a235_bb1eb0f5893c.slice/crio-3269af04ee5e20bd247444a1c5285080a88195bde787260ab19fa751cef7c5a7 WatchSource:0}: Error finding container 3269af04ee5e20bd247444a1c5285080a88195bde787260ab19fa751cef7c5a7: Status 404 returned error can't find the container with id 3269af04ee5e20bd247444a1c5285080a88195bde787260ab19fa751cef7c5a7
Jul 28 21:26:32.457031 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:32.456988    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v" event={"ID":"44811dcd-c92f-47cb-ba20-84af02416fc5","Type":"ContainerStarted","Data":"86e2daf47080d40d19dd23f77f738c5396a9a3157b65a1175bdf291a6e683e24"}
Jul 28 21:26:32.457532 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:32.457060    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v" event={"ID":"44811dcd-c92f-47cb-ba20-84af02416fc5","Type":"ContainerStarted","Data":"ded6bb56935118d612bd694a5bede4c595b97c39da128a14b70d3e4fe3412519"}
Jul 28 21:26:32.457532 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:32.457130    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v"
Jul 28 21:26:32.459686 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:32.459651    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj" event={"ID":"0a402b0c-5d3a-4081-a235-bb1eb0f5893c","Type":"ContainerStarted","Data":"fca4fbbb35fdeb56dacbc4591656ec27f1c06ead88518dd1ed14b54879cc704c"}
Jul 28 21:26:32.459847 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:32.459699    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj" event={"ID":"0a402b0c-5d3a-4081-a235-bb1eb0f5893c","Type":"ContainerStarted","Data":"3269af04ee5e20bd247444a1c5285080a88195bde787260ab19fa751cef7c5a7"}
Jul 28 21:26:32.459847 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:32.459730    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj"
Jul 28 21:26:32.476539 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:32.476486    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v" podStartSLOduration=1.476467529 podStartE2EDuration="1.476467529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:26:32.474057522 +0000 UTC m=+2347.698400510" watchObservedRunningTime="2026-07-28 21:26:32.476467529 +0000 UTC m=+2347.700810498"
Jul 28 21:26:32.495221 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:32.495165    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj" podStartSLOduration=1.495148951 podStartE2EDuration="1.495148951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:26:32.492317858 +0000 UTC m=+2347.716660828" watchObservedRunningTime="2026-07-28 21:26:32.495148951 +0000 UTC m=+2347.719491937"
Jul 28 21:26:34.474599 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:34.474560    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4" event={"ID":"8f4e9546-5762-426f-b651-d9cd5c9a7c20","Type":"ContainerStarted","Data":"07391eea0a1d25cc9707c7c7aa57a0e927e05dacc4a3757ca7437fef8d187edb"}
Jul 28 21:26:34.475193 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:34.474676    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"
Jul 28 21:26:34.495308 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:34.495260    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4" podStartSLOduration=1.9464323590000001 podStartE2EDuration="4.495246099s" totalImagesPullingTime="2.54881374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:26:34.4929253 +0000 UTC m=+2349.717268287" watchObservedRunningTime="2026-07-28 21:26:34.495246099 +0000 UTC m=+2349.719589085"
Jul 28 21:26:36.165261 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:36.165224    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:26:40.485616 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:40.485585    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"
Jul 28 21:26:43.468981 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:43.468949    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj"
Jul 28 21:26:43.469363 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:43.469108    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v"
Jul 28 21:26:58.007970 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:58.007939    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c4e74-zxnbj"]
Jul 28 21:26:58.012532 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:58.012512    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c4e74-zxnbj"
Jul 28 21:26:58.014288 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:58.014268    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w22xz\""
Jul 28 21:26:58.018576 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:58.018554    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c4e74-zxnbj"]
Jul 28 21:26:58.126526 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:58.126493    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c05cb9b-da45-489f-83a4-d4d99cab539f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3c4e74-zxnbj\" (UID: \"2c05cb9b-da45-489f-83a4-d4d99cab539f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c4e74-zxnbj"
Jul 28 21:26:58.227829 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:58.227788    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c05cb9b-da45-489f-83a4-d4d99cab539f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3c4e74-zxnbj\" (UID: \"2c05cb9b-da45-489f-83a4-d4d99cab539f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c4e74-zxnbj"
Jul 28 21:26:58.228442 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:58.228419    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c05cb9b-da45-489f-83a4-d4d99cab539f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3c4e74-zxnbj\" (UID: \"2c05cb9b-da45-489f-83a4-d4d99cab539f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c4e74-zxnbj"
Jul 28 21:26:58.324089 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:58.324002    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c4e74-zxnbj"
Jul 28 21:26:58.460753 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:58.460723    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c4e74-zxnbj"]
Jul 28 21:26:58.462778 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:26:58.462752    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c05cb9b_da45_489f_83a4_d4d99cab539f.slice/crio-f502077b337523f8cab486c896b6e223616ae901e49e0352f13c56575f35182e WatchSource:0}: Error finding container f502077b337523f8cab486c896b6e223616ae901e49e0352f13c56575f35182e: Status 404 returned error can't find the container with id f502077b337523f8cab486c896b6e223616ae901e49e0352f13c56575f35182e
Jul 28 21:26:58.608059 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:58.607982    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c4e74-zxnbj" event={"ID":"2c05cb9b-da45-489f-83a4-d4d99cab539f","Type":"ContainerStarted","Data":"f502077b337523f8cab486c896b6e223616ae901e49e0352f13c56575f35182e"}
Jul 28 21:26:59.615298 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:59.615263    2565 generic.go:356] "Generic (PLEG): container finished" podID="2c05cb9b-da45-489f-83a4-d4d99cab539f" containerID="242fb3f54f1301d8d144d0ac002b0105d77f9e6403cbbe30c7b7258edc0eec38" exitCode=0
Jul 28 21:26:59.615713 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:26:59.615306    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c4e74-zxnbj" event={"ID":"2c05cb9b-da45-489f-83a4-d4d99cab539f","Type":"ContainerDied","Data":"242fb3f54f1301d8d144d0ac002b0105d77f9e6403cbbe30c7b7258edc0eec38"}
Jul 28 21:27:00.753560 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:00.753537    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c4e74-zxnbj"
Jul 28 21:27:00.851327 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:00.851291    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c05cb9b-da45-489f-83a4-d4d99cab539f-maas-api-service-ca\") pod \"2c05cb9b-da45-489f-83a4-d4d99cab539f\" (UID: \"2c05cb9b-da45-489f-83a4-d4d99cab539f\") "
Jul 28 21:27:00.851676 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:00.851629    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c05cb9b-da45-489f-83a4-d4d99cab539f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c05cb9b-da45-489f-83a4-d4d99cab539f" (UID: "2c05cb9b-da45-489f-83a4-d4d99cab539f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:27:00.952519 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:00.952485    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c05cb9b-da45-489f-83a4-d4d99cab539f-maas-api-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:27:01.627750 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:01.627666    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c4e74-zxnbj"
Jul 28 21:27:01.627750 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:01.627683    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c4e74-zxnbj" event={"ID":"2c05cb9b-da45-489f-83a4-d4d99cab539f","Type":"ContainerDied","Data":"f502077b337523f8cab486c896b6e223616ae901e49e0352f13c56575f35182e"}
Jul 28 21:27:01.627750 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:01.627729    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f502077b337523f8cab486c896b6e223616ae901e49e0352f13c56575f35182e"
Jul 28 21:27:13.733616 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:13.733581    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k"]
Jul 28 21:27:13.734207 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:13.733861    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k" podUID="176b0935-7d01-4063-b179-725fd5995f91" containerName="payload-processing" containerID="cri-o://4e27debf4d331ab80fb4e2f312a25e0478036e43c382b9fffd2e0950210daa1c" gracePeriod=30
Jul 28 21:27:13.740775 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:13.740742    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9"]
Jul 28 21:27:13.742074 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:13.742016    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9" podUID="b1844b76-2ef1-470b-baa9-4a62d25f782f" containerName="payload-pre-processing" containerID="cri-o://d6b3e9590321881e3d124d7757723d0e793741809c62d47a0738043b58aa7c29" gracePeriod=30
Jul 28 21:27:13.868937 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:13.868899    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c4e74-zxnbj"]
Jul 28 21:27:13.871194 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:13.871165    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c4e74-zxnbj"]
Jul 28 21:27:14.020777 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.020751    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k"
Jul 28 21:27:14.044822 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.044788    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9"
Jul 28 21:27:14.174765 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.174733    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1844b76-2ef1-470b-baa9-4a62d25f782f-plugins-config-volume\") pod \"b1844b76-2ef1-470b-baa9-4a62d25f782f\" (UID: \"b1844b76-2ef1-470b-baa9-4a62d25f782f\") "
Jul 28 21:27:14.174966 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.174791    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvbgl\" (UniqueName: \"kubernetes.io/projected/b1844b76-2ef1-470b-baa9-4a62d25f782f-kube-api-access-qvbgl\") pod \"b1844b76-2ef1-470b-baa9-4a62d25f782f\" (UID: \"b1844b76-2ef1-470b-baa9-4a62d25f782f\") "
Jul 28 21:27:14.174966 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.174860    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wfql\" (UniqueName: \"kubernetes.io/projected/176b0935-7d01-4063-b179-725fd5995f91-kube-api-access-7wfql\") pod \"176b0935-7d01-4063-b179-725fd5995f91\" (UID: \"176b0935-7d01-4063-b179-725fd5995f91\") "
Jul 28 21:27:14.174966 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.174899    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/176b0935-7d01-4063-b179-725fd5995f91-plugins-config-volume\") pod \"176b0935-7d01-4063-b179-725fd5995f91\" (UID: \"176b0935-7d01-4063-b179-725fd5995f91\") "
Jul 28 21:27:14.175144 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.175123    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1844b76-2ef1-470b-baa9-4a62d25f782f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b1844b76-2ef1-470b-baa9-4a62d25f782f" (UID: "b1844b76-2ef1-470b-baa9-4a62d25f782f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:27:14.175349 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.175326    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176b0935-7d01-4063-b179-725fd5995f91-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "176b0935-7d01-4063-b179-725fd5995f91" (UID: "176b0935-7d01-4063-b179-725fd5995f91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:27:14.176941 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.176921    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1844b76-2ef1-470b-baa9-4a62d25f782f-kube-api-access-qvbgl" (OuterVolumeSpecName: "kube-api-access-qvbgl") pod "b1844b76-2ef1-470b-baa9-4a62d25f782f" (UID: "b1844b76-2ef1-470b-baa9-4a62d25f782f"). InnerVolumeSpecName "kube-api-access-qvbgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:27:14.176996 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.176926    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176b0935-7d01-4063-b179-725fd5995f91-kube-api-access-7wfql" (OuterVolumeSpecName: "kube-api-access-7wfql") pod "176b0935-7d01-4063-b179-725fd5995f91" (UID: "176b0935-7d01-4063-b179-725fd5995f91"). InnerVolumeSpecName "kube-api-access-7wfql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:27:14.275849 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.275766    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wfql\" (UniqueName: \"kubernetes.io/projected/176b0935-7d01-4063-b179-725fd5995f91-kube-api-access-7wfql\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:27:14.275849 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.275793    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/176b0935-7d01-4063-b179-725fd5995f91-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:27:14.275849 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.275803    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1844b76-2ef1-470b-baa9-4a62d25f782f-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:27:14.275849 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.275812    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvbgl\" (UniqueName: \"kubernetes.io/projected/b1844b76-2ef1-470b-baa9-4a62d25f782f-kube-api-access-qvbgl\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:27:14.706280 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.706251    2565 generic.go:356] "Generic (PLEG): container finished" podID="176b0935-7d01-4063-b179-725fd5995f91" containerID="4e27debf4d331ab80fb4e2f312a25e0478036e43c382b9fffd2e0950210daa1c" exitCode=0
Jul 28 21:27:14.706489 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.706301    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k"
Jul 28 21:27:14.706489 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.706348    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k" event={"ID":"176b0935-7d01-4063-b179-725fd5995f91","Type":"ContainerDied","Data":"4e27debf4d331ab80fb4e2f312a25e0478036e43c382b9fffd2e0950210daa1c"}
Jul 28 21:27:14.706489 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.706400    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k" event={"ID":"176b0935-7d01-4063-b179-725fd5995f91","Type":"ContainerDied","Data":"a691869040ff7c718bc9834da811d389fffa1835865ef87f2d7c99a7a4f27c76"}
Jul 28 21:27:14.706489 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.706422    2565 scope.go:117] "RemoveContainer" containerID="4e27debf4d331ab80fb4e2f312a25e0478036e43c382b9fffd2e0950210daa1c"
Jul 28 21:27:14.707926 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.707899    2565 generic.go:356] "Generic (PLEG): container finished" podID="b1844b76-2ef1-470b-baa9-4a62d25f782f" containerID="d6b3e9590321881e3d124d7757723d0e793741809c62d47a0738043b58aa7c29" exitCode=0
Jul 28 21:27:14.708044 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.707944    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9"
Jul 28 21:27:14.708044 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.707951    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9" event={"ID":"b1844b76-2ef1-470b-baa9-4a62d25f782f","Type":"ContainerDied","Data":"d6b3e9590321881e3d124d7757723d0e793741809c62d47a0738043b58aa7c29"}
Jul 28 21:27:14.708044 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.707986    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9" event={"ID":"b1844b76-2ef1-470b-baa9-4a62d25f782f","Type":"ContainerDied","Data":"19e56745f04cdd0a7a3bf6736cf538e765e6d6cf7399ed0a88272d184f7bc19d"}
Jul 28 21:27:14.716181 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.716158    2565 scope.go:117] "RemoveContainer" containerID="4e27debf4d331ab80fb4e2f312a25e0478036e43c382b9fffd2e0950210daa1c"
Jul 28 21:27:14.716501 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:27:14.716476    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e27debf4d331ab80fb4e2f312a25e0478036e43c382b9fffd2e0950210daa1c\": container with ID starting with 4e27debf4d331ab80fb4e2f312a25e0478036e43c382b9fffd2e0950210daa1c not found: ID does not exist" containerID="4e27debf4d331ab80fb4e2f312a25e0478036e43c382b9fffd2e0950210daa1c"
Jul 28 21:27:14.716605 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.716512    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e27debf4d331ab80fb4e2f312a25e0478036e43c382b9fffd2e0950210daa1c"} err="failed to get container status \"4e27debf4d331ab80fb4e2f312a25e0478036e43c382b9fffd2e0950210daa1c\": rpc error: code = NotFound desc = could not find container \"4e27debf4d331ab80fb4e2f312a25e0478036e43c382b9fffd2e0950210daa1c\": container with ID starting with 4e27debf4d331ab80fb4e2f312a25e0478036e43c382b9fffd2e0950210daa1c not found: ID does not exist"
Jul 28 21:27:14.716605 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.716536    2565 scope.go:117] "RemoveContainer" containerID="d6b3e9590321881e3d124d7757723d0e793741809c62d47a0738043b58aa7c29"
Jul 28 21:27:14.727938 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.727915    2565 scope.go:117] "RemoveContainer" containerID="d6b3e9590321881e3d124d7757723d0e793741809c62d47a0738043b58aa7c29"
Jul 28 21:27:14.728207 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:27:14.728189    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b3e9590321881e3d124d7757723d0e793741809c62d47a0738043b58aa7c29\": container with ID starting with d6b3e9590321881e3d124d7757723d0e793741809c62d47a0738043b58aa7c29 not found: ID does not exist" containerID="d6b3e9590321881e3d124d7757723d0e793741809c62d47a0738043b58aa7c29"
Jul 28 21:27:14.728248 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.728218    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b3e9590321881e3d124d7757723d0e793741809c62d47a0738043b58aa7c29"} err="failed to get container status \"d6b3e9590321881e3d124d7757723d0e793741809c62d47a0738043b58aa7c29\": rpc error: code = NotFound desc = could not find container \"d6b3e9590321881e3d124d7757723d0e793741809c62d47a0738043b58aa7c29\": container with ID starting with d6b3e9590321881e3d124d7757723d0e793741809c62d47a0738043b58aa7c29 not found: ID does not exist"
Jul 28 21:27:14.731289 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.731266    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k"]
Jul 28 21:27:14.734650 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.734614    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3c4e74-946746dd-w2r8k"]
Jul 28 21:27:14.748596 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.748572    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9"]
Jul 28 21:27:14.754034 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:14.754013    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3c4e74-57747ccb68-7qts9"]
Jul 28 21:27:15.332170 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:15.332138    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="176b0935-7d01-4063-b179-725fd5995f91" path="/var/lib/kubelet/pods/176b0935-7d01-4063-b179-725fd5995f91/volumes"
Jul 28 21:27:15.332524 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:15.332509    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c05cb9b-da45-489f-83a4-d4d99cab539f" path="/var/lib/kubelet/pods/2c05cb9b-da45-489f-83a4-d4d99cab539f/volumes"
Jul 28 21:27:15.332870 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:15.332857    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1844b76-2ef1-470b-baa9-4a62d25f782f" path="/var/lib/kubelet/pods/b1844b76-2ef1-470b-baa9-4a62d25f782f/volumes"
Jul 28 21:27:20.061380 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:20.061342    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:27:20.709747 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:20.709715    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65cc6c-r98gr"]
Jul 28 21:27:20.710065 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:20.710053    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c05cb9b-da45-489f-83a4-d4d99cab539f" containerName="revoke-keys"
Jul 28 21:27:20.710111 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:20.710067    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c05cb9b-da45-489f-83a4-d4d99cab539f" containerName="revoke-keys"
Jul 28 21:27:20.710111 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:20.710081    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1844b76-2ef1-470b-baa9-4a62d25f782f" containerName="payload-pre-processing"
Jul 28 21:27:20.710111 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:20.710087    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1844b76-2ef1-470b-baa9-4a62d25f782f" containerName="payload-pre-processing"
Jul 28 21:27:20.710111 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:20.710101    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="176b0935-7d01-4063-b179-725fd5995f91" containerName="payload-processing"
Jul 28 21:27:20.710111 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:20.710107    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="176b0935-7d01-4063-b179-725fd5995f91" containerName="payload-processing"
Jul 28 21:27:20.710272 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:20.710160    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1844b76-2ef1-470b-baa9-4a62d25f782f" containerName="payload-pre-processing"
Jul 28 21:27:20.710272 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:20.710172    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c05cb9b-da45-489f-83a4-d4d99cab539f" containerName="revoke-keys"
Jul 28 21:27:20.710272 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:20.710178    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="176b0935-7d01-4063-b179-725fd5995f91" containerName="payload-processing"
Jul 28 21:27:20.714823 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:20.714805    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65cc6c-r98gr"
Jul 28 21:27:20.716747 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:20.716727    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w22xz\""
Jul 28 21:27:20.720915 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:20.720888    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65cc6c-r98gr"]
Jul 28 21:27:20.835024 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:20.834986    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/390f049a-91af-4911-ac4d-a42f9e56da73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-65cc6c-r98gr\" (UID: \"390f049a-91af-4911-ac4d-a42f9e56da73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65cc6c-r98gr"
Jul 28 21:27:20.936355 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:20.936302    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/390f049a-91af-4911-ac4d-a42f9e56da73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-65cc6c-r98gr\" (UID: \"390f049a-91af-4911-ac4d-a42f9e56da73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65cc6c-r98gr"
Jul 28 21:27:20.937052 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:20.937027    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/390f049a-91af-4911-ac4d-a42f9e56da73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-65cc6c-r98gr\" (UID: \"390f049a-91af-4911-ac4d-a42f9e56da73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65cc6c-r98gr"
Jul 28 21:27:21.027326 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:21.027219    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65cc6c-r98gr"
Jul 28 21:27:21.177522 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:21.177498    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65cc6c-r98gr"]
Jul 28 21:27:21.179678 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:27:21.179645    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390f049a_91af_4911_ac4d_a42f9e56da73.slice/crio-2c140d13700028895b05620236df81afca8e8bfa66e6dbe4adae47e1cc468c83 WatchSource:0}: Error finding container 2c140d13700028895b05620236df81afca8e8bfa66e6dbe4adae47e1cc468c83: Status 404 returned error can't find the container with id 2c140d13700028895b05620236df81afca8e8bfa66e6dbe4adae47e1cc468c83
Jul 28 21:27:21.751462 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:21.751428    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65cc6c-r98gr" event={"ID":"390f049a-91af-4911-ac4d-a42f9e56da73","Type":"ContainerStarted","Data":"33df9741a3a7924982b2a4586b186e60f38bc0cc4b325bd1e28ce70ecfbe8f19"}
Jul 28 21:27:21.751688 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:21.751483    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65cc6c-r98gr" event={"ID":"390f049a-91af-4911-ac4d-a42f9e56da73","Type":"ContainerStarted","Data":"2c140d13700028895b05620236df81afca8e8bfa66e6dbe4adae47e1cc468c83"}
Jul 28 21:27:21.767098 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:21.767055    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65cc6c-r98gr" podStartSLOduration=1.767040874 podStartE2EDuration="1.767040874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:27:21.76633173 +0000 UTC m=+2396.990674717" watchObservedRunningTime="2026-07-28 21:27:21.767040874 +0000 UTC m=+2396.991383860"
Jul 28 21:27:22.758420 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:22.758386    2565 generic.go:356] "Generic (PLEG): container finished" podID="390f049a-91af-4911-ac4d-a42f9e56da73" containerID="33df9741a3a7924982b2a4586b186e60f38bc0cc4b325bd1e28ce70ecfbe8f19" exitCode=0
Jul 28 21:27:22.758894 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:22.758469    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65cc6c-r98gr" event={"ID":"390f049a-91af-4911-ac4d-a42f9e56da73","Type":"ContainerDied","Data":"33df9741a3a7924982b2a4586b186e60f38bc0cc4b325bd1e28ce70ecfbe8f19"}
Jul 28 21:27:23.893855 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:23.893827    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65cc6c-r98gr"
Jul 28 21:27:24.063317 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:24.063223    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/390f049a-91af-4911-ac4d-a42f9e56da73-maas-api-service-ca\") pod \"390f049a-91af-4911-ac4d-a42f9e56da73\" (UID: \"390f049a-91af-4911-ac4d-a42f9e56da73\") "
Jul 28 21:27:24.063611 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:24.063585    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390f049a-91af-4911-ac4d-a42f9e56da73-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "390f049a-91af-4911-ac4d-a42f9e56da73" (UID: "390f049a-91af-4911-ac4d-a42f9e56da73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:27:24.164614 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:24.164572    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/390f049a-91af-4911-ac4d-a42f9e56da73-maas-api-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:27:24.771189 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:24.771157    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65cc6c-r98gr"
Jul 28 21:27:24.771189 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:24.771173    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65cc6c-r98gr" event={"ID":"390f049a-91af-4911-ac4d-a42f9e56da73","Type":"ContainerDied","Data":"2c140d13700028895b05620236df81afca8e8bfa66e6dbe4adae47e1cc468c83"}
Jul 28 21:27:24.771412 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:24.771225    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c140d13700028895b05620236df81afca8e8bfa66e6dbe4adae47e1cc468c83"
Jul 28 21:27:27.298794 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:27.298760    2565 scope.go:117] "RemoveContainer" containerID="0c255d87755c6de255df0ba87ba7dafca227fdf28d8bb2fbeb64f4cbf819c798"
Jul 28 21:27:30.809505 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:30.809472    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"]
Jul 28 21:27:30.810026 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:30.809748    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4" podUID="8f4e9546-5762-426f-b651-d9cd5c9a7c20" containerName="maas-api" containerID="cri-o://07391eea0a1d25cc9707c7c7aa57a0e927e05dacc4a3757ca7437fef8d187edb" gracePeriod=30
Jul 28 21:27:30.868410 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:30.868377    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj"]
Jul 28 21:27:30.868625 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:30.868582    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj" podUID="0a402b0c-5d3a-4081-a235-bb1eb0f5893c" containerName="payload-processing" containerID="cri-o://fca4fbbb35fdeb56dacbc4591656ec27f1c06ead88518dd1ed14b54879cc704c" gracePeriod=30
Jul 28 21:27:30.876042 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:30.876009    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v"]
Jul 28 21:27:30.876245 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:30.876222    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v" podUID="44811dcd-c92f-47cb-ba20-84af02416fc5" containerName="payload-pre-processing" containerID="cri-o://86e2daf47080d40d19dd23f77f738c5396a9a3157b65a1175bdf291a6e683e24" gracePeriod=30
Jul 28 21:27:31.027397 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.027327    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65cc6c-r98gr"]
Jul 28 21:27:31.030900 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.030859    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-65cc6c-r98gr"]
Jul 28 21:27:31.145110 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.145082    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"
Jul 28 21:27:31.151770 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.151743    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj"
Jul 28 21:27:31.161058 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.161038    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v"
Jul 28 21:27:31.327964 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.327919    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44811dcd-c92f-47cb-ba20-84af02416fc5-plugins-config-volume\") pod \"44811dcd-c92f-47cb-ba20-84af02416fc5\" (UID: \"44811dcd-c92f-47cb-ba20-84af02416fc5\") "
Jul 28 21:27:31.328176 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.327983    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4e9546-5762-426f-b651-d9cd5c9a7c20-openshift-service-ca\") pod \"8f4e9546-5762-426f-b651-d9cd5c9a7c20\" (UID: \"8f4e9546-5762-426f-b651-d9cd5c9a7c20\") "
Jul 28 21:27:31.328176 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.328018    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdgl5\" (UniqueName: \"kubernetes.io/projected/44811dcd-c92f-47cb-ba20-84af02416fc5-kube-api-access-qdgl5\") pod \"44811dcd-c92f-47cb-ba20-84af02416fc5\" (UID: \"44811dcd-c92f-47cb-ba20-84af02416fc5\") "
Jul 28 21:27:31.328176 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.328164    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4e9546-5762-426f-b651-d9cd5c9a7c20-maas-api-tls\") pod \"8f4e9546-5762-426f-b651-d9cd5c9a7c20\" (UID: \"8f4e9546-5762-426f-b651-d9cd5c9a7c20\") "
Jul 28 21:27:31.328371 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.328208    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlkst\" (UniqueName: \"kubernetes.io/projected/8f4e9546-5762-426f-b651-d9cd5c9a7c20-kube-api-access-wlkst\") pod \"8f4e9546-5762-426f-b651-d9cd5c9a7c20\" (UID: \"8f4e9546-5762-426f-b651-d9cd5c9a7c20\") "
Jul 28 21:27:31.328371 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.328241    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a402b0c-5d3a-4081-a235-bb1eb0f5893c-plugins-config-volume\") pod \"0a402b0c-5d3a-4081-a235-bb1eb0f5893c\" (UID: \"0a402b0c-5d3a-4081-a235-bb1eb0f5893c\") "
Jul 28 21:27:31.328371 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.328321    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h98rj\" (UniqueName: \"kubernetes.io/projected/0a402b0c-5d3a-4081-a235-bb1eb0f5893c-kube-api-access-h98rj\") pod \"0a402b0c-5d3a-4081-a235-bb1eb0f5893c\" (UID: \"0a402b0c-5d3a-4081-a235-bb1eb0f5893c\") "
Jul 28 21:27:31.328371 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.328361    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4e9546-5762-426f-b651-d9cd5c9a7c20-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8f4e9546-5762-426f-b651-d9cd5c9a7c20" (UID: "8f4e9546-5762-426f-b651-d9cd5c9a7c20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:27:31.328597 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.328365    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44811dcd-c92f-47cb-ba20-84af02416fc5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "44811dcd-c92f-47cb-ba20-84af02416fc5" (UID: "44811dcd-c92f-47cb-ba20-84af02416fc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:27:31.328686 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.328591    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44811dcd-c92f-47cb-ba20-84af02416fc5-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:27:31.328686 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.328612    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4e9546-5762-426f-b651-d9cd5c9a7c20-openshift-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:27:31.328794 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.328737    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a402b0c-5d3a-4081-a235-bb1eb0f5893c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a402b0c-5d3a-4081-a235-bb1eb0f5893c" (UID: "0a402b0c-5d3a-4081-a235-bb1eb0f5893c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:27:31.330505 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.330477    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44811dcd-c92f-47cb-ba20-84af02416fc5-kube-api-access-qdgl5" (OuterVolumeSpecName: "kube-api-access-qdgl5") pod "44811dcd-c92f-47cb-ba20-84af02416fc5" (UID: "44811dcd-c92f-47cb-ba20-84af02416fc5"). InnerVolumeSpecName "kube-api-access-qdgl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:27:31.330797 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.330572    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f4e9546-5762-426f-b651-d9cd5c9a7c20-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8f4e9546-5762-426f-b651-d9cd5c9a7c20" (UID: "8f4e9546-5762-426f-b651-d9cd5c9a7c20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:27:31.330797 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.330609    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4e9546-5762-426f-b651-d9cd5c9a7c20-kube-api-access-wlkst" (OuterVolumeSpecName: "kube-api-access-wlkst") pod "8f4e9546-5762-426f-b651-d9cd5c9a7c20" (UID: "8f4e9546-5762-426f-b651-d9cd5c9a7c20"). InnerVolumeSpecName "kube-api-access-wlkst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:27:31.330909 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.330862    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a402b0c-5d3a-4081-a235-bb1eb0f5893c-kube-api-access-h98rj" (OuterVolumeSpecName: "kube-api-access-h98rj") pod "0a402b0c-5d3a-4081-a235-bb1eb0f5893c" (UID: "0a402b0c-5d3a-4081-a235-bb1eb0f5893c"). InnerVolumeSpecName "kube-api-access-h98rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:27:31.332486 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.332464    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="390f049a-91af-4911-ac4d-a42f9e56da73" path="/var/lib/kubelet/pods/390f049a-91af-4911-ac4d-a42f9e56da73/volumes"
Jul 28 21:27:31.429288 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.429249    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdgl5\" (UniqueName: \"kubernetes.io/projected/44811dcd-c92f-47cb-ba20-84af02416fc5-kube-api-access-qdgl5\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:27:31.429288 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.429280    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4e9546-5762-426f-b651-d9cd5c9a7c20-maas-api-tls\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:27:31.429288 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.429292    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlkst\" (UniqueName: \"kubernetes.io/projected/8f4e9546-5762-426f-b651-d9cd5c9a7c20-kube-api-access-wlkst\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:27:31.429526 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.429301    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a402b0c-5d3a-4081-a235-bb1eb0f5893c-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:27:31.429526 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.429310    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h98rj\" (UniqueName: \"kubernetes.io/projected/0a402b0c-5d3a-4081-a235-bb1eb0f5893c-kube-api-access-h98rj\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:27:31.811440 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.811405    2565 generic.go:356] "Generic (PLEG): container finished" podID="44811dcd-c92f-47cb-ba20-84af02416fc5" containerID="86e2daf47080d40d19dd23f77f738c5396a9a3157b65a1175bdf291a6e683e24" exitCode=0
Jul 28 21:27:31.811901 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.811449    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v"
Jul 28 21:27:31.811901 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.811498    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v" event={"ID":"44811dcd-c92f-47cb-ba20-84af02416fc5","Type":"ContainerDied","Data":"86e2daf47080d40d19dd23f77f738c5396a9a3157b65a1175bdf291a6e683e24"}
Jul 28 21:27:31.811901 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.811558    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v" event={"ID":"44811dcd-c92f-47cb-ba20-84af02416fc5","Type":"ContainerDied","Data":"ded6bb56935118d612bd694a5bede4c595b97c39da128a14b70d3e4fe3412519"}
Jul 28 21:27:31.811901 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.811595    2565 scope.go:117] "RemoveContainer" containerID="86e2daf47080d40d19dd23f77f738c5396a9a3157b65a1175bdf291a6e683e24"
Jul 28 21:27:31.812962 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.812900    2565 generic.go:356] "Generic (PLEG): container finished" podID="0a402b0c-5d3a-4081-a235-bb1eb0f5893c" containerID="fca4fbbb35fdeb56dacbc4591656ec27f1c06ead88518dd1ed14b54879cc704c" exitCode=0
Jul 28 21:27:31.812962 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.812948    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj"
Jul 28 21:27:31.813087 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.812997    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj" event={"ID":"0a402b0c-5d3a-4081-a235-bb1eb0f5893c","Type":"ContainerDied","Data":"fca4fbbb35fdeb56dacbc4591656ec27f1c06ead88518dd1ed14b54879cc704c"}
Jul 28 21:27:31.813087 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.813051    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj" event={"ID":"0a402b0c-5d3a-4081-a235-bb1eb0f5893c","Type":"ContainerDied","Data":"3269af04ee5e20bd247444a1c5285080a88195bde787260ab19fa751cef7c5a7"}
Jul 28 21:27:31.814442 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.814421    2565 generic.go:356] "Generic (PLEG): container finished" podID="8f4e9546-5762-426f-b651-d9cd5c9a7c20" containerID="07391eea0a1d25cc9707c7c7aa57a0e927e05dacc4a3757ca7437fef8d187edb" exitCode=0
Jul 28 21:27:31.814537 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.814482    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4" event={"ID":"8f4e9546-5762-426f-b651-d9cd5c9a7c20","Type":"ContainerDied","Data":"07391eea0a1d25cc9707c7c7aa57a0e927e05dacc4a3757ca7437fef8d187edb"}
Jul 28 21:27:31.814537 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.814516    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"
Jul 28 21:27:31.814537 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.814523    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4" event={"ID":"8f4e9546-5762-426f-b651-d9cd5c9a7c20","Type":"ContainerDied","Data":"2f57602e833dc890963ccabfba57e427d1216c6428985813aa2a66d467e4f373"}
Jul 28 21:27:31.822069 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.822053    2565 scope.go:117] "RemoveContainer" containerID="86e2daf47080d40d19dd23f77f738c5396a9a3157b65a1175bdf291a6e683e24"
Jul 28 21:27:31.822324 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:27:31.822304    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e2daf47080d40d19dd23f77f738c5396a9a3157b65a1175bdf291a6e683e24\": container with ID starting with 86e2daf47080d40d19dd23f77f738c5396a9a3157b65a1175bdf291a6e683e24 not found: ID does not exist" containerID="86e2daf47080d40d19dd23f77f738c5396a9a3157b65a1175bdf291a6e683e24"
Jul 28 21:27:31.822374 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.822331    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e2daf47080d40d19dd23f77f738c5396a9a3157b65a1175bdf291a6e683e24"} err="failed to get container status \"86e2daf47080d40d19dd23f77f738c5396a9a3157b65a1175bdf291a6e683e24\": rpc error: code = NotFound desc = could not find container \"86e2daf47080d40d19dd23f77f738c5396a9a3157b65a1175bdf291a6e683e24\": container with ID starting with 86e2daf47080d40d19dd23f77f738c5396a9a3157b65a1175bdf291a6e683e24 not found: ID does not exist"
Jul 28 21:27:31.822374 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.822348    2565 scope.go:117] "RemoveContainer" containerID="fca4fbbb35fdeb56dacbc4591656ec27f1c06ead88518dd1ed14b54879cc704c"
Jul 28 21:27:31.835734 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.835715    2565 scope.go:117] "RemoveContainer" containerID="fca4fbbb35fdeb56dacbc4591656ec27f1c06ead88518dd1ed14b54879cc704c"
Jul 28 21:27:31.836001 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:27:31.835981    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca4fbbb35fdeb56dacbc4591656ec27f1c06ead88518dd1ed14b54879cc704c\": container with ID starting with fca4fbbb35fdeb56dacbc4591656ec27f1c06ead88518dd1ed14b54879cc704c not found: ID does not exist" containerID="fca4fbbb35fdeb56dacbc4591656ec27f1c06ead88518dd1ed14b54879cc704c"
Jul 28 21:27:31.836068 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.836007    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca4fbbb35fdeb56dacbc4591656ec27f1c06ead88518dd1ed14b54879cc704c"} err="failed to get container status \"fca4fbbb35fdeb56dacbc4591656ec27f1c06ead88518dd1ed14b54879cc704c\": rpc error: code = NotFound desc = could not find container \"fca4fbbb35fdeb56dacbc4591656ec27f1c06ead88518dd1ed14b54879cc704c\": container with ID starting with fca4fbbb35fdeb56dacbc4591656ec27f1c06ead88518dd1ed14b54879cc704c not found: ID does not exist"
Jul 28 21:27:31.836068 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.836025    2565 scope.go:117] "RemoveContainer" containerID="07391eea0a1d25cc9707c7c7aa57a0e927e05dacc4a3757ca7437fef8d187edb"
Jul 28 21:27:31.836759 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.836740    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj"]
Jul 28 21:27:31.841424 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.841401    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65cc6c-5f4dfdbc5d-glwzj"]
Jul 28 21:27:31.844457 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.844437    2565 scope.go:117] "RemoveContainer" containerID="07391eea0a1d25cc9707c7c7aa57a0e927e05dacc4a3757ca7437fef8d187edb"
Jul 28 21:27:31.844733 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:27:31.844707    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07391eea0a1d25cc9707c7c7aa57a0e927e05dacc4a3757ca7437fef8d187edb\": container with ID starting with 07391eea0a1d25cc9707c7c7aa57a0e927e05dacc4a3757ca7437fef8d187edb not found: ID does not exist" containerID="07391eea0a1d25cc9707c7c7aa57a0e927e05dacc4a3757ca7437fef8d187edb"
Jul 28 21:27:31.844829 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.844737    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07391eea0a1d25cc9707c7c7aa57a0e927e05dacc4a3757ca7437fef8d187edb"} err="failed to get container status \"07391eea0a1d25cc9707c7c7aa57a0e927e05dacc4a3757ca7437fef8d187edb\": rpc error: code = NotFound desc = could not find container \"07391eea0a1d25cc9707c7c7aa57a0e927e05dacc4a3757ca7437fef8d187edb\": container with ID starting with 07391eea0a1d25cc9707c7c7aa57a0e927e05dacc4a3757ca7437fef8d187edb not found: ID does not exist"
Jul 28 21:27:31.851789 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.851768    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v"]
Jul 28 21:27:31.855786 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.855766    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65cc6c-64cfbd549b-l242v"]
Jul 28 21:27:31.865092 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.865056    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"]
Jul 28 21:27:31.867134 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:31.867116    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-65cc6c-8549d8b847-s6mv4"]
Jul 28 21:27:33.057163 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:33.057123    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:27:33.332364 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:33.332271    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a402b0c-5d3a-4081-a235-bb1eb0f5893c" path="/var/lib/kubelet/pods/0a402b0c-5d3a-4081-a235-bb1eb0f5893c/volumes"
Jul 28 21:27:33.332680 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:33.332664    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44811dcd-c92f-47cb-ba20-84af02416fc5" path="/var/lib/kubelet/pods/44811dcd-c92f-47cb-ba20-84af02416fc5/volumes"
Jul 28 21:27:33.333022 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:33.333009    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f4e9546-5762-426f-b651-d9cd5c9a7c20" path="/var/lib/kubelet/pods/8f4e9546-5762-426f-b651-d9cd5c9a7c20/volumes"
Jul 28 21:27:41.800682 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.800651    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"]
Jul 28 21:27:41.801080 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.800996    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="390f049a-91af-4911-ac4d-a42f9e56da73" containerName="revoke-keys"
Jul 28 21:27:41.801080 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.801010    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="390f049a-91af-4911-ac4d-a42f9e56da73" containerName="revoke-keys"
Jul 28 21:27:41.801080 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.801034    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44811dcd-c92f-47cb-ba20-84af02416fc5" containerName="payload-pre-processing"
Jul 28 21:27:41.801080 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.801045    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="44811dcd-c92f-47cb-ba20-84af02416fc5" containerName="payload-pre-processing"
Jul 28 21:27:41.801080 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.801052    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f4e9546-5762-426f-b651-d9cd5c9a7c20" containerName="maas-api"
Jul 28 21:27:41.801080 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.801057    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f4e9546-5762-426f-b651-d9cd5c9a7c20" containerName="maas-api"
Jul 28 21:27:41.801080 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.801072    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a402b0c-5d3a-4081-a235-bb1eb0f5893c" containerName="payload-processing"
Jul 28 21:27:41.801080 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.801077    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a402b0c-5d3a-4081-a235-bb1eb0f5893c" containerName="payload-processing"
Jul 28 21:27:41.801341 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.801135    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a402b0c-5d3a-4081-a235-bb1eb0f5893c" containerName="payload-processing"
Jul 28 21:27:41.801341 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.801144    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="44811dcd-c92f-47cb-ba20-84af02416fc5" containerName="payload-pre-processing"
Jul 28 21:27:41.801341 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.801150    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="390f049a-91af-4911-ac4d-a42f9e56da73" containerName="revoke-keys"
Jul 28 21:27:41.801341 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.801157    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f4e9546-5762-426f-b651-d9cd5c9a7c20" containerName="maas-api"
Jul 28 21:27:41.805248 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.805227    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"
Jul 28 21:27:41.807655 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.807619    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-1096ca-serving-cert\""
Jul 28 21:27:41.807804 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.807689    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2thqx\""
Jul 28 21:27:41.807804 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.807720    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 21:27:41.808942 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.808922    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 21:27:41.815537 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.815388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhb9w\" (UniqueName: \"kubernetes.io/projected/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-kube-api-access-hhb9w\") pod \"maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k\" (UID: \"e258c9f3-c7eb-46a2-8c03-98bb9f2bda41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"
Jul 28 21:27:41.815537 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.815453    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k\" (UID: \"e258c9f3-c7eb-46a2-8c03-98bb9f2bda41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"
Jul 28 21:27:41.815537 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.815502    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-maas-api-tls\") pod \"maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k\" (UID: \"e258c9f3-c7eb-46a2-8c03-98bb9f2bda41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"
Jul 28 21:27:41.815946 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.815922    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"]
Jul 28 21:27:41.916503 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.916458    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhb9w\" (UniqueName: \"kubernetes.io/projected/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-kube-api-access-hhb9w\") pod \"maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k\" (UID: \"e258c9f3-c7eb-46a2-8c03-98bb9f2bda41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"
Jul 28 21:27:41.916688 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.916522    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k\" (UID: \"e258c9f3-c7eb-46a2-8c03-98bb9f2bda41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"
Jul 28 21:27:41.916688 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.916567    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-maas-api-tls\") pod \"maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k\" (UID: \"e258c9f3-c7eb-46a2-8c03-98bb9f2bda41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"
Jul 28 21:27:41.916791 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:27:41.916693    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-serving-cert: secret "maas-api-e2e-shared-a-1096ca-serving-cert" not found
Jul 28 21:27:41.916791 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:27:41.916778    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-maas-api-tls podName:e258c9f3-c7eb-46a2-8c03-98bb9f2bda41 nodeName:}" failed. No retries permitted until 2026-07-28 21:27:42.416763052 +0000 UTC m=+2417.641106016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-maas-api-tls") pod "maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k" (UID: "e258c9f3-c7eb-46a2-8c03-98bb9f2bda41") : secret "maas-api-e2e-shared-a-1096ca-serving-cert" not found
Jul 28 21:27:41.917198 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.917179    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k\" (UID: \"e258c9f3-c7eb-46a2-8c03-98bb9f2bda41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"
Jul 28 21:27:41.932849 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:41.932820    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhb9w\" (UniqueName: \"kubernetes.io/projected/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-kube-api-access-hhb9w\") pod \"maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k\" (UID: \"e258c9f3-c7eb-46a2-8c03-98bb9f2bda41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"
Jul 28 21:27:42.421507 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.421466    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-maas-api-tls\") pod \"maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k\" (UID: \"e258c9f3-c7eb-46a2-8c03-98bb9f2bda41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"
Jul 28 21:27:42.423905 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.423874    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-maas-api-tls\") pod \"maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k\" (UID: \"e258c9f3-c7eb-46a2-8c03-98bb9f2bda41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"
Jul 28 21:27:42.652773 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.652737    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b"]
Jul 28 21:27:42.656402 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.656386    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b"
Jul 28 21:27:42.658248 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.658221    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-1096ca\""
Jul 28 21:27:42.658384 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.658330    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-1096ca-dockercfg-dwvn2\""
Jul 28 21:27:42.668997 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.668975    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b"]
Jul 28 21:27:42.684915 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.684895    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2"]
Jul 28 21:27:42.688250 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.688235    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2"
Jul 28 21:27:42.698517 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.698493    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2"]
Jul 28 21:27:42.716737 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.716713    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"
Jul 28 21:27:42.723809 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.723787    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqvh2\" (UniqueName: \"kubernetes.io/projected/c60268ae-e839-4cb8-8b7c-422893332578-kube-api-access-fqvh2\") pod \"payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b\" (UID: \"c60268ae-e839-4cb8-8b7c-422893332578\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b"
Jul 28 21:27:42.723907 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.723826    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9slr\" (UniqueName: \"kubernetes.io/projected/8a9ac520-5c17-4a81-a614-8e597ba74942-kube-api-access-h9slr\") pod \"payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2\" (UID: \"8a9ac520-5c17-4a81-a614-8e597ba74942\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2"
Jul 28 21:27:42.723907 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.723863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c60268ae-e839-4cb8-8b7c-422893332578-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b\" (UID: \"c60268ae-e839-4cb8-8b7c-422893332578\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b"
Jul 28 21:27:42.723907 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.723905    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9ac520-5c17-4a81-a614-8e597ba74942-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2\" (UID: \"8a9ac520-5c17-4a81-a614-8e597ba74942\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2"
Jul 28 21:27:42.825351 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.825314    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c60268ae-e839-4cb8-8b7c-422893332578-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b\" (UID: \"c60268ae-e839-4cb8-8b7c-422893332578\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b"
Jul 28 21:27:42.825778 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.825359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9ac520-5c17-4a81-a614-8e597ba74942-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2\" (UID: \"8a9ac520-5c17-4a81-a614-8e597ba74942\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2"
Jul 28 21:27:42.825778 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.825400    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqvh2\" (UniqueName: \"kubernetes.io/projected/c60268ae-e839-4cb8-8b7c-422893332578-kube-api-access-fqvh2\") pod \"payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b\" (UID: \"c60268ae-e839-4cb8-8b7c-422893332578\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b"
Jul 28 21:27:42.825778 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.825473    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9slr\" (UniqueName: \"kubernetes.io/projected/8a9ac520-5c17-4a81-a614-8e597ba74942-kube-api-access-h9slr\") pod \"payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2\" (UID: \"8a9ac520-5c17-4a81-a614-8e597ba74942\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2"
Jul 28 21:27:42.826330 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.826299    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9ac520-5c17-4a81-a614-8e597ba74942-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2\" (UID: \"8a9ac520-5c17-4a81-a614-8e597ba74942\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2"
Jul 28 21:27:42.826427 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.826322    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c60268ae-e839-4cb8-8b7c-422893332578-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b\" (UID: \"c60268ae-e839-4cb8-8b7c-422893332578\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b"
Jul 28 21:27:42.833428 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.833399    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqvh2\" (UniqueName: \"kubernetes.io/projected/c60268ae-e839-4cb8-8b7c-422893332578-kube-api-access-fqvh2\") pod \"payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b\" (UID: \"c60268ae-e839-4cb8-8b7c-422893332578\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b"
Jul 28 21:27:42.833535 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.833497    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9slr\" (UniqueName: \"kubernetes.io/projected/8a9ac520-5c17-4a81-a614-8e597ba74942-kube-api-access-h9slr\") pod \"payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2\" (UID: \"8a9ac520-5c17-4a81-a614-8e597ba74942\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2"
Jul 28 21:27:42.852971 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.852945    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"]
Jul 28 21:27:42.854729 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:27:42.854705    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode258c9f3_c7eb_46a2_8c03_98bb9f2bda41.slice/crio-7c89ed9d2c772473729270214e5535a85571d1d49879d80d04541f0c7051e110 WatchSource:0}: Error finding container 7c89ed9d2c772473729270214e5535a85571d1d49879d80d04541f0c7051e110: Status 404 returned error can't find the container with id 7c89ed9d2c772473729270214e5535a85571d1d49879d80d04541f0c7051e110
Jul 28 21:27:42.877265 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.877232    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k" event={"ID":"e258c9f3-c7eb-46a2-8c03-98bb9f2bda41","Type":"ContainerStarted","Data":"7c89ed9d2c772473729270214e5535a85571d1d49879d80d04541f0c7051e110"}
Jul 28 21:27:42.966841 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.966750    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b"
Jul 28 21:27:42.998808 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:42.998776    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2"
Jul 28 21:27:43.118818 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:43.118572    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b"]
Jul 28 21:27:43.121256 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:27:43.121227    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60268ae_e839_4cb8_8b7c_422893332578.slice/crio-ca76d0e5350ea0266a7d0fe348ed7bf48e6158af260c2810d62045f1def66f4d WatchSource:0}: Error finding container ca76d0e5350ea0266a7d0fe348ed7bf48e6158af260c2810d62045f1def66f4d: Status 404 returned error can't find the container with id ca76d0e5350ea0266a7d0fe348ed7bf48e6158af260c2810d62045f1def66f4d
Jul 28 21:27:43.152584 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:43.152562    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2"]
Jul 28 21:27:43.154690 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:27:43.154664    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9ac520_5c17_4a81_a614_8e597ba74942.slice/crio-10a98d3fa14f3f33e589af889a8fd63c6755d5bb02aeba4ea83ffa1f26c93562 WatchSource:0}: Error finding container 10a98d3fa14f3f33e589af889a8fd63c6755d5bb02aeba4ea83ffa1f26c93562: Status 404 returned error can't find the container with id 10a98d3fa14f3f33e589af889a8fd63c6755d5bb02aeba4ea83ffa1f26c93562
Jul 28 21:27:43.884576 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:43.884535    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k" event={"ID":"e258c9f3-c7eb-46a2-8c03-98bb9f2bda41","Type":"ContainerStarted","Data":"a6536e41e8c7d88802e68ff15a4c5b52055a16c9f809c83f8a87e5e8cf8846a1"}
Jul 28 21:27:43.885043 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:43.884677    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"
Jul 28 21:27:43.886156 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:43.886128    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2" event={"ID":"8a9ac520-5c17-4a81-a614-8e597ba74942","Type":"ContainerStarted","Data":"70973c36453aa4cc1e232276a94d1b6f8f28db706965c2114624ba9a3ad2d07d"}
Jul 28 21:27:43.886267 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:43.886178    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2"
Jul 28 21:27:43.886267 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:43.886197    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2" event={"ID":"8a9ac520-5c17-4a81-a614-8e597ba74942","Type":"ContainerStarted","Data":"10a98d3fa14f3f33e589af889a8fd63c6755d5bb02aeba4ea83ffa1f26c93562"}
Jul 28 21:27:43.887645 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:43.887614    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b" event={"ID":"c60268ae-e839-4cb8-8b7c-422893332578","Type":"ContainerStarted","Data":"2ada3a5faa22b4ebae4b004bff65c16dc8c0d8e1d140678561d4d0ec9e38c0db"}
Jul 28 21:27:43.887733 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:43.887664    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b" event={"ID":"c60268ae-e839-4cb8-8b7c-422893332578","Type":"ContainerStarted","Data":"ca76d0e5350ea0266a7d0fe348ed7bf48e6158af260c2810d62045f1def66f4d"}
Jul 28 21:27:43.887814 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:43.887804    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b"
Jul 28 21:27:43.902092 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:43.902039    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k" podStartSLOduration=2.538700215 podStartE2EDuration="2.902022009s" totalImagesPullingTime="363.321794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:27:43.899400941 +0000 UTC m=+2419.123743926" watchObservedRunningTime="2026-07-28 21:27:43.902022009 +0000 UTC m=+2419.126364995"
Jul 28 21:27:43.913611 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:43.913556    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b" podStartSLOduration=1.9135430119999999 podStartE2EDuration="1.913543012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:27:43.912146198 +0000 UTC m=+2419.136489187" watchObservedRunningTime="2026-07-28 21:27:43.913543012 +0000 UTC m=+2419.137885998"
Jul 28 21:27:43.926992 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:43.926938    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2" podStartSLOduration=1.926920813 podStartE2EDuration="1.926920813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:27:43.925574993 +0000 UTC m=+2419.149917979" watchObservedRunningTime="2026-07-28 21:27:43.926920813 +0000 UTC m=+2419.151263798"
Jul 28 21:27:45.754559 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:45.754520    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:27:49.898858 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:49.898778    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"
Jul 28 21:27:54.895452 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:54.895415    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2"
Jul 28 21:27:54.895902 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:27:54.895755    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b"
Jul 28 21:28:09.872100 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:09.872063    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"]
Jul 28 21:28:09.875519 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:09.875503    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"
Jul 28 21:28:09.877518 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:09.877498    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-2c3f70-serving-cert\""
Jul 28 21:28:09.885316 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:09.885296    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"]
Jul 28 21:28:09.960171 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:09.960131    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af83bbaf-5051-4c55-8d86-d6b7dc35790d-maas-api-tls\") pod \"maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf\" (UID: \"af83bbaf-5051-4c55-8d86-d6b7dc35790d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"
Jul 28 21:28:09.960371 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:09.960251    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af83bbaf-5051-4c55-8d86-d6b7dc35790d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf\" (UID: \"af83bbaf-5051-4c55-8d86-d6b7dc35790d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"
Jul 28 21:28:09.960371 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:09.960291    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsz7t\" (UniqueName: \"kubernetes.io/projected/af83bbaf-5051-4c55-8d86-d6b7dc35790d-kube-api-access-bsz7t\") pod \"maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf\" (UID: \"af83bbaf-5051-4c55-8d86-d6b7dc35790d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"
Jul 28 21:28:10.061478 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:10.061433    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af83bbaf-5051-4c55-8d86-d6b7dc35790d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf\" (UID: \"af83bbaf-5051-4c55-8d86-d6b7dc35790d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"
Jul 28 21:28:10.061478 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:10.061483    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsz7t\" (UniqueName: \"kubernetes.io/projected/af83bbaf-5051-4c55-8d86-d6b7dc35790d-kube-api-access-bsz7t\") pod \"maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf\" (UID: \"af83bbaf-5051-4c55-8d86-d6b7dc35790d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"
Jul 28 21:28:10.061812 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:10.061566    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af83bbaf-5051-4c55-8d86-d6b7dc35790d-maas-api-tls\") pod \"maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf\" (UID: \"af83bbaf-5051-4c55-8d86-d6b7dc35790d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"
Jul 28 21:28:10.061812 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:28:10.061734    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-serving-cert: secret "maas-api-e2e-shared-b-2c3f70-serving-cert" not found
Jul 28 21:28:10.061930 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:28:10.061820    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af83bbaf-5051-4c55-8d86-d6b7dc35790d-maas-api-tls podName:af83bbaf-5051-4c55-8d86-d6b7dc35790d nodeName:}" failed. No retries permitted until 2026-07-28 21:28:10.561796066 +0000 UTC m=+2445.786139044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af83bbaf-5051-4c55-8d86-d6b7dc35790d-maas-api-tls") pod "maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf" (UID: "af83bbaf-5051-4c55-8d86-d6b7dc35790d") : secret "maas-api-e2e-shared-b-2c3f70-serving-cert" not found
Jul 28 21:28:10.062188 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:10.062165    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af83bbaf-5051-4c55-8d86-d6b7dc35790d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf\" (UID: \"af83bbaf-5051-4c55-8d86-d6b7dc35790d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"
Jul 28 21:28:10.071566 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:10.071543    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsz7t\" (UniqueName: \"kubernetes.io/projected/af83bbaf-5051-4c55-8d86-d6b7dc35790d-kube-api-access-bsz7t\") pod \"maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf\" (UID: \"af83bbaf-5051-4c55-8d86-d6b7dc35790d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"
Jul 28 21:28:10.565256 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:10.565218    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af83bbaf-5051-4c55-8d86-d6b7dc35790d-maas-api-tls\") pod \"maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf\" (UID: \"af83bbaf-5051-4c55-8d86-d6b7dc35790d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"
Jul 28 21:28:10.567471 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:10.567437    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp"]
Jul 28 21:28:10.567849 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:10.567827    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af83bbaf-5051-4c55-8d86-d6b7dc35790d-maas-api-tls\") pod \"maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf\" (UID: \"af83bbaf-5051-4c55-8d86-d6b7dc35790d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"
Jul 28 21:28:10.571063 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:10.571044    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp"
Jul 28 21:28:10.572851 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:10.572829    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-2c3f70\""
Jul 28 21:28:10.578971 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:10.578947    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp"]
Jul 28 21:28:10.666354 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:10.666318    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43b4211-355b-4736-a310-712b4761f4db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp\" (UID: \"a43b4211-355b-4736-a310-712b4761f4db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp"
Jul 28 21:28:10.666551 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:10.666374    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-722gq\" (UniqueName: \"kubernetes.io/projected/a43b4211-355b-4736-a310-712b4761f4db-kube-api-access-722gq\") pod \"payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp\" (UID: \"a43b4211-355b-4736-a310-712b4761f4db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp"
Jul 28 21:28:10.767078 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:10.767036    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43b4211-355b-4736-a310-712b4761f4db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp\" (UID: \"a43b4211-355b-4736-a310-712b4761f4db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp"
Jul 28 21:28:10.767280 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:10.767106    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-722gq\" (UniqueName: \"kubernetes.io/projected/a43b4211-355b-4736-a310-712b4761f4db-kube-api-access-722gq\") pod \"payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp\" (UID: \"a43b4211-355b-4736-a310-712b4761f4db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp"
Jul 28 21:28:10.767765 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:10.767746    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43b4211-355b-4736-a310-712b4761f4db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp\" (UID: \"a43b4211-355b-4736-a310-712b4761f4db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp"
Jul 28 21:28:10.775004 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:10.774974    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-722gq\" (UniqueName: \"kubernetes.io/projected/a43b4211-355b-4736-a310-712b4761f4db-kube-api-access-722gq\") pod \"payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp\" (UID: \"a43b4211-355b-4736-a310-712b4761f4db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp"
Jul 28 21:28:10.786650 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:10.786619    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"
Jul 28 21:28:10.885973 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:10.885941    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp"
Jul 28 21:28:10.925839 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:10.925807    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"]
Jul 28 21:28:10.927260 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:28:10.927212    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf83bbaf_5051_4c55_8d86_d6b7dc35790d.slice/crio-05b1463535b8f130543b47edcf38a42c9a3baad36bf586b244c2803340b1d9c2 WatchSource:0}: Error finding container 05b1463535b8f130543b47edcf38a42c9a3baad36bf586b244c2803340b1d9c2: Status 404 returned error can't find the container with id 05b1463535b8f130543b47edcf38a42c9a3baad36bf586b244c2803340b1d9c2
Jul 28 21:28:11.025512 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:11.025481    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp"]
Jul 28 21:28:11.027310 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:28:11.027285    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43b4211_355b_4736_a310_712b4761f4db.slice/crio-c2b7bdf1894968937daa7faac9bbd58cdc34e460c70e43914b8f12c3a6236bf9 WatchSource:0}: Error finding container c2b7bdf1894968937daa7faac9bbd58cdc34e460c70e43914b8f12c3a6236bf9: Status 404 returned error can't find the container with id c2b7bdf1894968937daa7faac9bbd58cdc34e460c70e43914b8f12c3a6236bf9
Jul 28 21:28:11.050824 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:11.050792    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf" event={"ID":"af83bbaf-5051-4c55-8d86-d6b7dc35790d","Type":"ContainerStarted","Data":"05b1463535b8f130543b47edcf38a42c9a3baad36bf586b244c2803340b1d9c2"}
Jul 28 21:28:11.052008 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:11.051986    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp" event={"ID":"a43b4211-355b-4736-a310-712b4761f4db","Type":"ContainerStarted","Data":"c2b7bdf1894968937daa7faac9bbd58cdc34e460c70e43914b8f12c3a6236bf9"}
Jul 28 21:28:12.058902 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:12.058863    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf" event={"ID":"af83bbaf-5051-4c55-8d86-d6b7dc35790d","Type":"ContainerStarted","Data":"3a93bd682b5429d99b8606b8bd9df1482975478cca2b35c505bff9882c887230"}
Jul 28 21:28:12.059407 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:12.058977    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"
Jul 28 21:28:12.060438 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:12.060416    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp" event={"ID":"a43b4211-355b-4736-a310-712b4761f4db","Type":"ContainerStarted","Data":"22d6faad98194f9765192cf614c9189a82ee46df4aadfb94f389cc4ff6be696d"}
Jul 28 21:28:12.060545 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:12.060470    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp"
Jul 28 21:28:12.076537 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:12.076488    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf" podStartSLOduration=2.728259726 podStartE2EDuration="3.076472061s" totalImagesPullingTime="348.212335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:28:12.074204554 +0000 UTC m=+2447.298547540" watchObservedRunningTime="2026-07-28 21:28:12.076472061 +0000 UTC m=+2447.300815048"
Jul 28 21:28:12.092401 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:12.092349    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp" podStartSLOduration=2.092331835 podStartE2EDuration="2.092331835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:28:12.090478471 +0000 UTC m=+2447.314821458" watchObservedRunningTime="2026-07-28 21:28:12.092331835 +0000 UTC m=+2447.316674798"
Jul 28 21:28:16.952139 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:16.952099    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:28:18.072050 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:18.072021    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"
Jul 28 21:28:23.068582 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:23.068554    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp"
Jul 28 21:28:47.263573 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:28:47.263538    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:29:23.060003 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:29:23.059926    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:29:46.208311 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:29:46.208266    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:30:00.135374 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.135341    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1096ca-29754570-4rpx2"]
Jul 28 21:30:00.139269 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.139245    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754570-2txlk"]
Jul 28 21:30:00.139421 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.139374    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1096ca-29754570-4rpx2"
Jul 28 21:30:00.141698 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.141677    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w22xz\""
Jul 28 21:30:00.142303 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.142287    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2c3f70-29754570-xbx5x"]
Jul 28 21:30:00.142423 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.142405    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754570-2txlk"
Jul 28 21:30:00.145431 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.145411    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2c3f70-29754570-xbx5x"
Jul 28 21:30:00.149501 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.149474    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1096ca-29754570-4rpx2"]
Jul 28 21:30:00.152972 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.152946    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754570-2txlk"]
Jul 28 21:30:00.153089 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.153028    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1096ca-29754570-4rpx2"
Jul 28 21:30:00.155796 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.155771    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2c3f70-29754570-xbx5x"]
Jul 28 21:30:00.162308 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.162286    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754570-2txlk"
Jul 28 21:30:00.198935 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.198912    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2c3f70-29754570-xbx5x"
Jul 28 21:30:00.315571 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.315537    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1096ca-29754570-4rpx2"]
Jul 28 21:30:00.316496 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:30:00.316472    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86108fef_e226_42c7_99f9_0b544415f8f0.slice/crio-3fc0316d7780c78d54a1abd576dcd5b96b5b6c38c9de430aa9d3c8eb881a0936 WatchSource:0}: Error finding container 3fc0316d7780c78d54a1abd576dcd5b96b5b6c38c9de430aa9d3c8eb881a0936: Status 404 returned error can't find the container with id 3fc0316d7780c78d54a1abd576dcd5b96b5b6c38c9de430aa9d3c8eb881a0936
Jul 28 21:30:00.343775 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.343747    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754570-2txlk"]
Jul 28 21:30:00.351917 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:30:00.351889    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc3fd82_4f33_4edf_b053_563b1ef08cae.slice/crio-0b6ca2f1c37dd1cbd7b4b5c0f039d7dfc4d3ad4b2880e98cf1455d03f706a6b2 WatchSource:0}: Error finding container 0b6ca2f1c37dd1cbd7b4b5c0f039d7dfc4d3ad4b2880e98cf1455d03f706a6b2: Status 404 returned error can't find the container with id 0b6ca2f1c37dd1cbd7b4b5c0f039d7dfc4d3ad4b2880e98cf1455d03f706a6b2
Jul 28 21:30:00.373898 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.373875    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2c3f70-29754570-xbx5x"]
Jul 28 21:30:00.376824 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:30:00.376799    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60cc0afc_b1b3_4d76_9f61_1a5a270daea9.slice/crio-8824f74f5d9661d7c6d6f9b2286fa7264e93dc6d5d46bff024cb2985ac06e7d0 WatchSource:0}: Error finding container 8824f74f5d9661d7c6d6f9b2286fa7264e93dc6d5d46bff024cb2985ac06e7d0: Status 404 returned error can't find the container with id 8824f74f5d9661d7c6d6f9b2286fa7264e93dc6d5d46bff024cb2985ac06e7d0
Jul 28 21:30:00.667700 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.667661    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2c3f70-29754570-xbx5x" event={"ID":"60cc0afc-b1b3-4d76-9f61-1a5a270daea9","Type":"ContainerStarted","Data":"0fc0e40278628b801d63d3cd2aaa2a3ba5119a928dc99d94ed76e75c27e04385"}
Jul 28 21:30:00.667909 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.667758    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2c3f70-29754570-xbx5x" event={"ID":"60cc0afc-b1b3-4d76-9f61-1a5a270daea9","Type":"ContainerStarted","Data":"8824f74f5d9661d7c6d6f9b2286fa7264e93dc6d5d46bff024cb2985ac06e7d0"}
Jul 28 21:30:00.669445 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.669410    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754570-2txlk" event={"ID":"8cc3fd82-4f33-4edf-b053-563b1ef08cae","Type":"ContainerStarted","Data":"46e10696cd7096b40458ca9edf7fcc788d51e7bdfa3eb33e5d863c0644ec340c"}
Jul 28 21:30:00.669583 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.669464    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754570-2txlk" event={"ID":"8cc3fd82-4f33-4edf-b053-563b1ef08cae","Type":"ContainerStarted","Data":"0b6ca2f1c37dd1cbd7b4b5c0f039d7dfc4d3ad4b2880e98cf1455d03f706a6b2"}
Jul 28 21:30:00.671119 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.671086    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1096ca-29754570-4rpx2" event={"ID":"86108fef-e226-42c7-99f9-0b544415f8f0","Type":"ContainerStarted","Data":"360f39b4de2f7f7e8e1ca994be34c5aa5604f94d3f75b78a3b3cc1119a93e098"}
Jul 28 21:30:00.671204 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.671147    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1096ca-29754570-4rpx2" event={"ID":"86108fef-e226-42c7-99f9-0b544415f8f0","Type":"ContainerStarted","Data":"3fc0316d7780c78d54a1abd576dcd5b96b5b6c38c9de430aa9d3c8eb881a0936"}
Jul 28 21:30:00.682461 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.682376    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2c3f70-29754570-xbx5x" podStartSLOduration=0.682360918 podStartE2EDuration="682.360918ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:30:00.681252203 +0000 UTC m=+2555.905595189" watchObservedRunningTime="2026-07-28 21:30:00.682360918 +0000 UTC m=+2555.906703882"
Jul 28 21:30:00.695594 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.695549    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1096ca-29754570-4rpx2" podStartSLOduration=0.6955354 podStartE2EDuration="695.5354ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:30:00.694435403 +0000 UTC m=+2555.918778402" watchObservedRunningTime="2026-07-28 21:30:00.6955354 +0000 UTC m=+2555.919878385"
Jul 28 21:30:00.709562 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:00.709510    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754570-2txlk" podStartSLOduration=0.709492597 podStartE2EDuration="709.492597ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:30:00.706989183 +0000 UTC m=+2555.931332170" watchObservedRunningTime="2026-07-28 21:30:00.709492597 +0000 UTC m=+2555.933835583"
Jul 28 21:30:01.679278 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:01.679243    2565 generic.go:356] "Generic (PLEG): container finished" podID="60cc0afc-b1b3-4d76-9f61-1a5a270daea9" containerID="0fc0e40278628b801d63d3cd2aaa2a3ba5119a928dc99d94ed76e75c27e04385" exitCode=0
Jul 28 21:30:01.679751 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:01.679287    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2c3f70-29754570-xbx5x" event={"ID":"60cc0afc-b1b3-4d76-9f61-1a5a270daea9","Type":"ContainerDied","Data":"0fc0e40278628b801d63d3cd2aaa2a3ba5119a928dc99d94ed76e75c27e04385"}
Jul 28 21:30:01.680903 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:01.680882    2565 generic.go:356] "Generic (PLEG): container finished" podID="8cc3fd82-4f33-4edf-b053-563b1ef08cae" containerID="46e10696cd7096b40458ca9edf7fcc788d51e7bdfa3eb33e5d863c0644ec340c" exitCode=0
Jul 28 21:30:01.681025 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:01.680965    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754570-2txlk" event={"ID":"8cc3fd82-4f33-4edf-b053-563b1ef08cae","Type":"ContainerDied","Data":"46e10696cd7096b40458ca9edf7fcc788d51e7bdfa3eb33e5d863c0644ec340c"}
Jul 28 21:30:01.682263 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:01.682247    2565 generic.go:356] "Generic (PLEG): container finished" podID="86108fef-e226-42c7-99f9-0b544415f8f0" containerID="360f39b4de2f7f7e8e1ca994be34c5aa5604f94d3f75b78a3b3cc1119a93e098" exitCode=0
Jul 28 21:30:01.682342 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:01.682308    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1096ca-29754570-4rpx2" event={"ID":"86108fef-e226-42c7-99f9-0b544415f8f0","Type":"ContainerDied","Data":"360f39b4de2f7f7e8e1ca994be34c5aa5604f94d3f75b78a3b3cc1119a93e098"}
Jul 28 21:30:02.833293 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:02.833268    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1096ca-29754570-4rpx2"
Jul 28 21:30:02.877750 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:02.877728    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754570-2txlk"
Jul 28 21:30:02.881287 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:02.881268    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2c3f70-29754570-xbx5x"
Jul 28 21:30:03.694726 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:03.694699    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2c3f70-29754570-xbx5x"
Jul 28 21:30:03.694907 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:03.694698    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2c3f70-29754570-xbx5x" event={"ID":"60cc0afc-b1b3-4d76-9f61-1a5a270daea9","Type":"ContainerDied","Data":"8824f74f5d9661d7c6d6f9b2286fa7264e93dc6d5d46bff024cb2985ac06e7d0"}
Jul 28 21:30:03.694907 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:03.694829    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8824f74f5d9661d7c6d6f9b2286fa7264e93dc6d5d46bff024cb2985ac06e7d0"
Jul 28 21:30:03.696029 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:03.696009    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754570-2txlk"
Jul 28 21:30:03.696172 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:03.696014    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754570-2txlk" event={"ID":"8cc3fd82-4f33-4edf-b053-563b1ef08cae","Type":"ContainerDied","Data":"0b6ca2f1c37dd1cbd7b4b5c0f039d7dfc4d3ad4b2880e98cf1455d03f706a6b2"}
Jul 28 21:30:03.696172 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:03.696118    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b6ca2f1c37dd1cbd7b4b5c0f039d7dfc4d3ad4b2880e98cf1455d03f706a6b2"
Jul 28 21:30:03.697414 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:03.697400    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1096ca-29754570-4rpx2"
Jul 28 21:30:03.697494 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:03.697443    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1096ca-29754570-4rpx2" event={"ID":"86108fef-e226-42c7-99f9-0b544415f8f0","Type":"ContainerDied","Data":"3fc0316d7780c78d54a1abd576dcd5b96b5b6c38c9de430aa9d3c8eb881a0936"}
Jul 28 21:30:03.697494 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:03.697474    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fc0316d7780c78d54a1abd576dcd5b96b5b6c38c9de430aa9d3c8eb881a0936"
Jul 28 21:30:06.954411 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:06.954372    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:30:25.060835 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:25.060799    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:30:52.065423 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:30:52.065324    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:31:05.456772 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:31:05.456734    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:31:17.655995 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:31:17.655963    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:31:27.369761 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:31:27.369725    2565 scope.go:117] "RemoveContainer" containerID="0e1b2c6211c37743f702348df63964c6c26e9a06dab211e690a861e0cd3171d9"
Jul 28 21:31:32.854245 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:31:32.854204    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:31:50.658725 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:31:50.658690    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:32:11.052561 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:32:11.052523    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:32:27.390779 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:32:27.390699    2565 scope.go:117] "RemoveContainer" containerID="acc51019bce1d89b1f2967acc6f8487a38a4193bbd626c137cc2f44fd70ded49"
Jul 28 21:32:30.166460 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:32:30.166414    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:32:51.887228 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:32:51.887185    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:33:09.952041 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:33:09.952006    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:33:27.412235 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:33:27.412203    2565 scope.go:117] "RemoveContainer" containerID="242fb3f54f1301d8d144d0ac002b0105d77f9e6403cbbe30c7b7258edc0eec38"
Jul 28 21:33:27.420923 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:33:27.420898    2565 scope.go:117] "RemoveContainer" containerID="33df9741a3a7924982b2a4586b186e60f38bc0cc4b325bd1e28ce70ecfbe8f19"
Jul 28 21:33:31.360129 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:33:31.360096    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:33:48.554554 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:33:48.554475    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:34:02.362820 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:02.362783    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:34:39.572537 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.572495    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"]
Jul 28 21:34:39.573183 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.573067    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60cc0afc-b1b3-4d76-9f61-1a5a270daea9" containerName="cleanup"
Jul 28 21:34:39.573183 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.573091    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="60cc0afc-b1b3-4d76-9f61-1a5a270daea9" containerName="cleanup"
Jul 28 21:34:39.573183 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.573106    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86108fef-e226-42c7-99f9-0b544415f8f0" containerName="cleanup"
Jul 28 21:34:39.573183 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.573115    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="86108fef-e226-42c7-99f9-0b544415f8f0" containerName="cleanup"
Jul 28 21:34:39.573183 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.573141    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cc3fd82-4f33-4edf-b053-563b1ef08cae" containerName="cleanup"
Jul 28 21:34:39.573183 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.573150    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cc3fd82-4f33-4edf-b053-563b1ef08cae" containerName="cleanup"
Jul 28 21:34:39.573497 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.573233    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cc3fd82-4f33-4edf-b053-563b1ef08cae" containerName="cleanup"
Jul 28 21:34:39.573497 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.573248    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="86108fef-e226-42c7-99f9-0b544415f8f0" containerName="cleanup"
Jul 28 21:34:39.573497 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.573259    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="60cc0afc-b1b3-4d76-9f61-1a5a270daea9" containerName="cleanup"
Jul 28 21:34:39.576522 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.576499    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:39.578459 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.578426    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2c3f70\"/\"sub-test-model-2c3f70-kserve-self-signed-certs\""
Jul 28 21:34:39.578459 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.578426    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2c3f70\"/\"kube-root-ca.crt\""
Jul 28 21:34:39.578624 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.578490    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2c3f70\"/\"default-dockercfg-cvrsq\""
Jul 28 21:34:39.578624 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.578517    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2c3f70\"/\"openshift-service-ca.crt\""
Jul 28 21:34:39.586606 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.586581    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"]
Jul 28 21:34:39.674914 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.674876    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-tmp-dir\") pod \"sub-test-model-2c3f70-kserve-78758899c-z5wgj\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") " pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:39.675075 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.674931    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdvcw\" (UniqueName: \"kubernetes.io/projected/d4b35fcf-39b1-438d-81b8-8dcbd7681267-kube-api-access-mdvcw\") pod \"sub-test-model-2c3f70-kserve-78758899c-z5wgj\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") " pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:39.675075 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.675019    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-model-cache\") pod \"sub-test-model-2c3f70-kserve-78758899c-z5wgj\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") " pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:39.675156 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.675089    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-home\") pod \"sub-test-model-2c3f70-kserve-78758899c-z5wgj\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") " pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:39.675156 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.675128    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b35fcf-39b1-438d-81b8-8dcbd7681267-tls-certs\") pod \"sub-test-model-2c3f70-kserve-78758899c-z5wgj\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") " pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:39.675222 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.675158    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-dshm\") pod \"sub-test-model-2c3f70-kserve-78758899c-z5wgj\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") " pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:39.775993 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.775957    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b35fcf-39b1-438d-81b8-8dcbd7681267-tls-certs\") pod \"sub-test-model-2c3f70-kserve-78758899c-z5wgj\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") " pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:39.776186 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.776002    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-dshm\") pod \"sub-test-model-2c3f70-kserve-78758899c-z5wgj\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") " pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:39.776186 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.776114    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-tmp-dir\") pod \"sub-test-model-2c3f70-kserve-78758899c-z5wgj\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") " pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:39.776324 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.776255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdvcw\" (UniqueName: \"kubernetes.io/projected/d4b35fcf-39b1-438d-81b8-8dcbd7681267-kube-api-access-mdvcw\") pod \"sub-test-model-2c3f70-kserve-78758899c-z5wgj\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") " pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:39.776324 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.776303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-model-cache\") pod \"sub-test-model-2c3f70-kserve-78758899c-z5wgj\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") " pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:39.776434 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.776339    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-home\") pod \"sub-test-model-2c3f70-kserve-78758899c-z5wgj\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") " pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:39.776545 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.776514    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-tmp-dir\") pod \"sub-test-model-2c3f70-kserve-78758899c-z5wgj\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") " pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:39.776707 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.776627    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-home\") pod \"sub-test-model-2c3f70-kserve-78758899c-z5wgj\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") " pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:39.776781 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.776714    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-model-cache\") pod \"sub-test-model-2c3f70-kserve-78758899c-z5wgj\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") " pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:39.778279 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.778260    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-dshm\") pod \"sub-test-model-2c3f70-kserve-78758899c-z5wgj\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") " pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:39.778469 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.778452    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b35fcf-39b1-438d-81b8-8dcbd7681267-tls-certs\") pod \"sub-test-model-2c3f70-kserve-78758899c-z5wgj\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") " pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:39.784312 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.784292    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdvcw\" (UniqueName: \"kubernetes.io/projected/d4b35fcf-39b1-438d-81b8-8dcbd7681267-kube-api-access-mdvcw\") pod \"sub-test-model-2c3f70-kserve-78758899c-z5wgj\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") " pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:39.886869 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:39.886772    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:40.020258 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:40.020225    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"]
Jul 28 21:34:40.022020 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:34:40.021990    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4b35fcf_39b1_438d_81b8_8dcbd7681267.slice/crio-9b0fe9f8148857ba51dbee9c08f12699d08e8473909fd220a4aa60a5d78d7971 WatchSource:0}: Error finding container 9b0fe9f8148857ba51dbee9c08f12699d08e8473909fd220a4aa60a5d78d7971: Status 404 returned error can't find the container with id 9b0fe9f8148857ba51dbee9c08f12699d08e8473909fd220a4aa60a5d78d7971
Jul 28 21:34:40.242982 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:40.242943    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj" event={"ID":"d4b35fcf-39b1-438d-81b8-8dcbd7681267","Type":"ContainerStarted","Data":"852e6d5ccbfc13a0059af7638d5dc50306d0449e02adf19f2b14a51201bcd7f3"}
Jul 28 21:34:40.242982 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:40.242990    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj" event={"ID":"d4b35fcf-39b1-438d-81b8-8dcbd7681267","Type":"ContainerStarted","Data":"9b0fe9f8148857ba51dbee9c08f12699d08e8473909fd220a4aa60a5d78d7971"}
Jul 28 21:34:40.261604 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:40.261550    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj" podStartSLOduration=1.26153406 podStartE2EDuration="1.26153406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:34:40.258783013 +0000 UTC m=+2835.483126000" watchObservedRunningTime="2026-07-28 21:34:40.26153406 +0000 UTC m=+2835.485877045"
Jul 28 21:34:42.958477 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:42.958439    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:34:49.887601 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:49.887566    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:49.888160 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:49.887620    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:49.902484 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:49.902455    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:34:50.312387 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:34:50.312360    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:35:11.258480 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:11.258442    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:35:30.156283 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:30.156187    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:35:37.157426 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:37.157390    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"]
Jul 28 21:35:47.457020 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:47.456985    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:35:52.178098 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.178051    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj" podUID="d4b35fcf-39b1-438d-81b8-8dcbd7681267" containerName="main" containerID="cri-o://852e6d5ccbfc13a0059af7638d5dc50306d0449e02adf19f2b14a51201bcd7f3" gracePeriod=45
Jul 28 21:35:52.424353 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.424327    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:35:52.426327 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.426308    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b35fcf-39b1-438d-81b8-8dcbd7681267-tls-certs\") pod \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") "
Jul 28 21:35:52.426432 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.426344    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-tmp-dir\") pod \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") "
Jul 28 21:35:52.426432 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.426371    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-dshm\") pod \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") "
Jul 28 21:35:52.426432 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.426390    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-model-cache\") pod \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") "
Jul 28 21:35:52.426596 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.426547    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdvcw\" (UniqueName: \"kubernetes.io/projected/d4b35fcf-39b1-438d-81b8-8dcbd7681267-kube-api-access-mdvcw\") pod \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") "
Jul 28 21:35:52.426684 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.426609    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4b35fcf-39b1-438d-81b8-8dcbd7681267" (UID: "d4b35fcf-39b1-438d-81b8-8dcbd7681267"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:35:52.426684 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.426616    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-home\") pod \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\" (UID: \"d4b35fcf-39b1-438d-81b8-8dcbd7681267\") "
Jul 28 21:35:52.426684 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.426630    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4b35fcf-39b1-438d-81b8-8dcbd7681267" (UID: "d4b35fcf-39b1-438d-81b8-8dcbd7681267"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:35:52.426862 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.426821    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-home" (OuterVolumeSpecName: "home") pod "d4b35fcf-39b1-438d-81b8-8dcbd7681267" (UID: "d4b35fcf-39b1-438d-81b8-8dcbd7681267"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:35:52.426918 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.426900    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-tmp-dir\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:35:52.426962 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.426924    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-model-cache\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:35:52.426962 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.426936    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-home\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:35:52.428757 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.428702    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-dshm" (OuterVolumeSpecName: "dshm") pod "d4b35fcf-39b1-438d-81b8-8dcbd7681267" (UID: "d4b35fcf-39b1-438d-81b8-8dcbd7681267"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:35:52.428757 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.428735    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b35fcf-39b1-438d-81b8-8dcbd7681267-kube-api-access-mdvcw" (OuterVolumeSpecName: "kube-api-access-mdvcw") pod "d4b35fcf-39b1-438d-81b8-8dcbd7681267" (UID: "d4b35fcf-39b1-438d-81b8-8dcbd7681267"). InnerVolumeSpecName "kube-api-access-mdvcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:35:52.428914 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.428775    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b35fcf-39b1-438d-81b8-8dcbd7681267-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4b35fcf-39b1-438d-81b8-8dcbd7681267" (UID: "d4b35fcf-39b1-438d-81b8-8dcbd7681267"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:35:52.527526 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.527492    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b35fcf-39b1-438d-81b8-8dcbd7681267-tls-certs\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:35:52.527526 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.527520    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b35fcf-39b1-438d-81b8-8dcbd7681267-dshm\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:35:52.527526 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.527535    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdvcw\" (UniqueName: \"kubernetes.io/projected/d4b35fcf-39b1-438d-81b8-8dcbd7681267-kube-api-access-mdvcw\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:35:52.638395 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.638356    2565 generic.go:356] "Generic (PLEG): container finished" podID="d4b35fcf-39b1-438d-81b8-8dcbd7681267" containerID="852e6d5ccbfc13a0059af7638d5dc50306d0449e02adf19f2b14a51201bcd7f3" exitCode=0
Jul 28 21:35:52.638395 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.638397    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"
Jul 28 21:35:52.638669 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.638438    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj" event={"ID":"d4b35fcf-39b1-438d-81b8-8dcbd7681267","Type":"ContainerDied","Data":"852e6d5ccbfc13a0059af7638d5dc50306d0449e02adf19f2b14a51201bcd7f3"}
Jul 28 21:35:52.638669 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.638475    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj" event={"ID":"d4b35fcf-39b1-438d-81b8-8dcbd7681267","Type":"ContainerDied","Data":"9b0fe9f8148857ba51dbee9c08f12699d08e8473909fd220a4aa60a5d78d7971"}
Jul 28 21:35:52.638669 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.638494    2565 scope.go:117] "RemoveContainer" containerID="852e6d5ccbfc13a0059af7638d5dc50306d0449e02adf19f2b14a51201bcd7f3"
Jul 28 21:35:52.648231 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.648212    2565 scope.go:117] "RemoveContainer" containerID="852e6d5ccbfc13a0059af7638d5dc50306d0449e02adf19f2b14a51201bcd7f3"
Jul 28 21:35:52.648494 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:35:52.648475    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"852e6d5ccbfc13a0059af7638d5dc50306d0449e02adf19f2b14a51201bcd7f3\": container with ID starting with 852e6d5ccbfc13a0059af7638d5dc50306d0449e02adf19f2b14a51201bcd7f3 not found: ID does not exist" containerID="852e6d5ccbfc13a0059af7638d5dc50306d0449e02adf19f2b14a51201bcd7f3"
Jul 28 21:35:52.648567 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.648510    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"852e6d5ccbfc13a0059af7638d5dc50306d0449e02adf19f2b14a51201bcd7f3"} err="failed to get container status \"852e6d5ccbfc13a0059af7638d5dc50306d0449e02adf19f2b14a51201bcd7f3\": rpc error: code = NotFound desc = could not find container \"852e6d5ccbfc13a0059af7638d5dc50306d0449e02adf19f2b14a51201bcd7f3\": container with ID starting with 852e6d5ccbfc13a0059af7638d5dc50306d0449e02adf19f2b14a51201bcd7f3 not found: ID does not exist"
Jul 28 21:35:52.677525 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.677490    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"]
Jul 28 21:35:52.697461 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:52.697431    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-2c3f70/sub-test-model-2c3f70-kserve-78758899c-z5wgj"]
Jul 28 21:35:53.332147 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:35:53.332113    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4b35fcf-39b1-438d-81b8-8dcbd7681267" path="/var/lib/kubelet/pods/d4b35fcf-39b1-438d-81b8-8dcbd7681267/volumes"
Jul 28 21:36:11.456568 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:11.456531    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:36:45.169314 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.169278    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"]
Jul 28 21:36:45.169749 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.169661    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b35fcf-39b1-438d-81b8-8dcbd7681267" containerName="main"
Jul 28 21:36:45.169749 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.169674    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b35fcf-39b1-438d-81b8-8dcbd7681267" containerName="main"
Jul 28 21:36:45.169749 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.169747    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b35fcf-39b1-438d-81b8-8dcbd7681267" containerName="main"
Jul 28 21:36:45.172799 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.172782    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:45.179580 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.179543    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-model-cache\") pod \"rate-test-model-2c3f70-kserve-588f9785f9-b58k9\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") " pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:45.179809 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.179779    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-tmp-dir\") pod \"rate-test-model-2c3f70-kserve-588f9785f9-b58k9\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") " pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:45.179901 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.179811    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86da8170-d97d-490b-92aa-15e27f6aa7ae-tls-certs\") pod \"rate-test-model-2c3f70-kserve-588f9785f9-b58k9\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") " pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:45.179901 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.179832    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-home\") pod \"rate-test-model-2c3f70-kserve-588f9785f9-b58k9\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") " pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:45.179987 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.179905    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-dshm\") pod \"rate-test-model-2c3f70-kserve-588f9785f9-b58k9\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") " pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:45.179987 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.179945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxv54\" (UniqueName: \"kubernetes.io/projected/86da8170-d97d-490b-92aa-15e27f6aa7ae-kube-api-access-dxv54\") pod \"rate-test-model-2c3f70-kserve-588f9785f9-b58k9\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") " pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:45.181336 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.181311    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2c3f70\"/\"rate-test-model-2c3f70-kserve-self-signed-certs\""
Jul 28 21:36:45.181590 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.181572    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2c3f70\"/\"kube-root-ca.crt\""
Jul 28 21:36:45.181844 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.181824    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2c3f70\"/\"openshift-service-ca.crt\""
Jul 28 21:36:45.182048 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.182025    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2c3f70\"/\"default-dockercfg-cvrsq\""
Jul 28 21:36:45.184524 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.184504    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"]
Jul 28 21:36:45.281087 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.281041    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-model-cache\") pod \"rate-test-model-2c3f70-kserve-588f9785f9-b58k9\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") " pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:45.281087 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.281089    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-tmp-dir\") pod \"rate-test-model-2c3f70-kserve-588f9785f9-b58k9\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") " pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:45.281334 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.281105    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86da8170-d97d-490b-92aa-15e27f6aa7ae-tls-certs\") pod \"rate-test-model-2c3f70-kserve-588f9785f9-b58k9\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") " pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:45.281334 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.281124    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-home\") pod \"rate-test-model-2c3f70-kserve-588f9785f9-b58k9\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") " pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:45.281334 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.281164    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-dshm\") pod \"rate-test-model-2c3f70-kserve-588f9785f9-b58k9\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") " pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:45.281334 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.281188    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxv54\" (UniqueName: \"kubernetes.io/projected/86da8170-d97d-490b-92aa-15e27f6aa7ae-kube-api-access-dxv54\") pod \"rate-test-model-2c3f70-kserve-588f9785f9-b58k9\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") " pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:45.281559 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.281539    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-model-cache\") pod \"rate-test-model-2c3f70-kserve-588f9785f9-b58k9\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") " pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:45.281610 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.281551    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-tmp-dir\") pod \"rate-test-model-2c3f70-kserve-588f9785f9-b58k9\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") " pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:45.281610 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.281590    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-home\") pod \"rate-test-model-2c3f70-kserve-588f9785f9-b58k9\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") " pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:45.283506 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.283484    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-dshm\") pod \"rate-test-model-2c3f70-kserve-588f9785f9-b58k9\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") " pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:45.283773 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.283758    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86da8170-d97d-490b-92aa-15e27f6aa7ae-tls-certs\") pod \"rate-test-model-2c3f70-kserve-588f9785f9-b58k9\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") " pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:45.293036 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.293016    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxv54\" (UniqueName: \"kubernetes.io/projected/86da8170-d97d-490b-92aa-15e27f6aa7ae-kube-api-access-dxv54\") pod \"rate-test-model-2c3f70-kserve-588f9785f9-b58k9\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") " pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:45.482728 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.482689    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:45.618692 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.618665    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"]
Jul 28 21:36:45.620320 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:36:45.620291    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86da8170_d97d_490b_92aa_15e27f6aa7ae.slice/crio-0fc59cd8d8ac8c4ab5f145a0902382d01faf68273832a313f0d3fa21b1b72eb6 WatchSource:0}: Error finding container 0fc59cd8d8ac8c4ab5f145a0902382d01faf68273832a313f0d3fa21b1b72eb6: Status 404 returned error can't find the container with id 0fc59cd8d8ac8c4ab5f145a0902382d01faf68273832a313f0d3fa21b1b72eb6
Jul 28 21:36:45.938241 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.938202    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9" event={"ID":"86da8170-d97d-490b-92aa-15e27f6aa7ae","Type":"ContainerStarted","Data":"74941fcdaabbd23ad5f93662b0e2f0408ebc3011ecf5d2dfdb7c798547c5bac6"}
Jul 28 21:36:45.938426 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.938254    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9" event={"ID":"86da8170-d97d-490b-92aa-15e27f6aa7ae","Type":"ContainerStarted","Data":"0fc59cd8d8ac8c4ab5f145a0902382d01faf68273832a313f0d3fa21b1b72eb6"}
Jul 28 21:36:45.954872 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:45.954812    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9" podStartSLOduration=0.954794681 podStartE2EDuration="954.794681ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:36:45.953070347 +0000 UTC m=+2961.177413334" watchObservedRunningTime="2026-07-28 21:36:45.954794681 +0000 UTC m=+2961.179137704"
Jul 28 21:36:48.853528 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:48.853440    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:36:55.483297 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:55.483258    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:55.483297 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:55.483309    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:55.495894 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:55.495868    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:36:56.004734 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:36:56.004701    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:37:20.857243 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:20.857208    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:37:35.455388 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:35.455358    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:37:49.958510 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:49.958471    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:37:54.745289 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:54.745252    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"]
Jul 28 21:37:55.686607 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:55.686571    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7298-140669728648240"]
Jul 28 21:37:55.690219 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:55.690202    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728648240"
Jul 28 21:37:55.692286 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:55.692256    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7xvrr\""
Jul 28 21:37:55.693589 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:55.693537    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7298-140669728648240"]
Jul 28 21:37:55.822196 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:55.822157    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzzm4\" (UniqueName: \"kubernetes.io/projected/9d1e484e-6617-4a66-85c9-efe0ab2c9848-kube-api-access-vzzm4\") pod \"test-curl-7298-140669728648240\" (UID: \"9d1e484e-6617-4a66-85c9-efe0ab2c9848\") " pod="opendatahub/test-curl-7298-140669728648240"
Jul 28 21:37:55.923145 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:55.923107    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzzm4\" (UniqueName: \"kubernetes.io/projected/9d1e484e-6617-4a66-85c9-efe0ab2c9848-kube-api-access-vzzm4\") pod \"test-curl-7298-140669728648240\" (UID: \"9d1e484e-6617-4a66-85c9-efe0ab2c9848\") " pod="opendatahub/test-curl-7298-140669728648240"
Jul 28 21:37:55.932235 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:55.932212    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzzm4\" (UniqueName: \"kubernetes.io/projected/9d1e484e-6617-4a66-85c9-efe0ab2c9848-kube-api-access-vzzm4\") pod \"test-curl-7298-140669728648240\" (UID: \"9d1e484e-6617-4a66-85c9-efe0ab2c9848\") " pod="opendatahub/test-curl-7298-140669728648240"
Jul 28 21:37:56.000496 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:56.000409    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728648240"
Jul 28 21:37:56.342578 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:56.342553    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7298-140669728648240"]
Jul 28 21:37:56.344514 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:37:56.344483    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d1e484e_6617_4a66_85c9_efe0ab2c9848.slice/crio-8632796d6ea118d2a3511536d2c1ab8530031f31d848b3b603050c0bf97dbc50 WatchSource:0}: Error finding container 8632796d6ea118d2a3511536d2c1ab8530031f31d848b3b603050c0bf97dbc50: Status 404 returned error can't find the container with id 8632796d6ea118d2a3511536d2c1ab8530031f31d848b3b603050c0bf97dbc50
Jul 28 21:37:56.345784 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:56.345762    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 21:37:57.320530 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:57.320443    2565 generic.go:356] "Generic (PLEG): container finished" podID="9d1e484e-6617-4a66-85c9-efe0ab2c9848" containerID="5220a3db73153ca6563bbc04298ca8264971504c1e2b62c44f0b8af1cb9cf327" exitCode=0
Jul 28 21:37:57.320937 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:57.320529    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7298-140669728648240" event={"ID":"9d1e484e-6617-4a66-85c9-efe0ab2c9848","Type":"ContainerDied","Data":"5220a3db73153ca6563bbc04298ca8264971504c1e2b62c44f0b8af1cb9cf327"}
Jul 28 21:37:57.320937 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:57.320568    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7298-140669728648240" event={"ID":"9d1e484e-6617-4a66-85c9-efe0ab2c9848","Type":"ContainerStarted","Data":"8632796d6ea118d2a3511536d2c1ab8530031f31d848b3b603050c0bf97dbc50"}
Jul 28 21:37:58.455463 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:58.455434    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728648240"
Jul 28 21:37:58.490602 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:58.490571    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7298-140669728648240_9d1e484e-6617-4a66-85c9-efe0ab2c9848/test-curl-7298-140669728648240/0.log"
Jul 28 21:37:58.512647 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:58.512608    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7298-140669728648240"]
Jul 28 21:37:58.518279 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:58.518258    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7298-140669728648240"]
Jul 28 21:37:58.547852 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:58.547830    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzzm4\" (UniqueName: \"kubernetes.io/projected/9d1e484e-6617-4a66-85c9-efe0ab2c9848-kube-api-access-vzzm4\") pod \"9d1e484e-6617-4a66-85c9-efe0ab2c9848\" (UID: \"9d1e484e-6617-4a66-85c9-efe0ab2c9848\") "
Jul 28 21:37:58.549960 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:58.549939    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1e484e-6617-4a66-85c9-efe0ab2c9848-kube-api-access-vzzm4" (OuterVolumeSpecName: "kube-api-access-vzzm4") pod "9d1e484e-6617-4a66-85c9-efe0ab2c9848" (UID: "9d1e484e-6617-4a66-85c9-efe0ab2c9848"). InnerVolumeSpecName "kube-api-access-vzzm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:37:58.640490 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:58.640409    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7298-140669728645680"]
Jul 28 21:37:58.640801 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:58.640785    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d1e484e-6617-4a66-85c9-efe0ab2c9848" containerName="test-curl-7298-140669728648240"
Jul 28 21:37:58.640849 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:58.640802    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d1e484e-6617-4a66-85c9-efe0ab2c9848" containerName="test-curl-7298-140669728648240"
Jul 28 21:37:58.640885 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:58.640873    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d1e484e-6617-4a66-85c9-efe0ab2c9848" containerName="test-curl-7298-140669728648240"
Jul 28 21:37:58.643882 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:58.643866    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728645680"
Jul 28 21:37:58.648648 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:58.648609    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzzm4\" (UniqueName: \"kubernetes.io/projected/9d1e484e-6617-4a66-85c9-efe0ab2c9848-kube-api-access-vzzm4\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:37:58.651127 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:58.651097    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7298-140669728645680"]
Jul 28 21:37:58.749722 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:58.749686    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmh5j\" (UniqueName: \"kubernetes.io/projected/1f252d24-58a1-49d5-be7b-310539ec92cf-kube-api-access-fmh5j\") pod \"test-curl-7298-140669728645680\" (UID: \"1f252d24-58a1-49d5-be7b-310539ec92cf\") " pod="opendatahub/test-curl-7298-140669728645680"
Jul 28 21:37:58.850908 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:58.850872    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmh5j\" (UniqueName: \"kubernetes.io/projected/1f252d24-58a1-49d5-be7b-310539ec92cf-kube-api-access-fmh5j\") pod \"test-curl-7298-140669728645680\" (UID: \"1f252d24-58a1-49d5-be7b-310539ec92cf\") " pod="opendatahub/test-curl-7298-140669728645680"
Jul 28 21:37:58.858453 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:58.858427    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmh5j\" (UniqueName: \"kubernetes.io/projected/1f252d24-58a1-49d5-be7b-310539ec92cf-kube-api-access-fmh5j\") pod \"test-curl-7298-140669728645680\" (UID: \"1f252d24-58a1-49d5-be7b-310539ec92cf\") " pod="opendatahub/test-curl-7298-140669728645680"
Jul 28 21:37:58.954792 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:58.954759    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728645680"
Jul 28 21:37:59.089795 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:59.089770    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7298-140669728645680"]
Jul 28 21:37:59.091745 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:37:59.091711    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f252d24_58a1_49d5_be7b_310539ec92cf.slice/crio-b9df9258044f93717072afa42a189afb75ddf6bfaf3661872948cce66b544834 WatchSource:0}: Error finding container b9df9258044f93717072afa42a189afb75ddf6bfaf3661872948cce66b544834: Status 404 returned error can't find the container with id b9df9258044f93717072afa42a189afb75ddf6bfaf3661872948cce66b544834
Jul 28 21:37:59.332103 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:59.332014    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728648240"
Jul 28 21:37:59.332103 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:59.332038    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d1e484e-6617-4a66-85c9-efe0ab2c9848" path="/var/lib/kubelet/pods/9d1e484e-6617-4a66-85c9-efe0ab2c9848/volumes"
Jul 28 21:37:59.332521 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:59.332503    2565 scope.go:117] "RemoveContainer" containerID="5220a3db73153ca6563bbc04298ca8264971504c1e2b62c44f0b8af1cb9cf327"
Jul 28 21:37:59.333238 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:37:59.333208    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7298-140669728645680" event={"ID":"1f252d24-58a1-49d5-be7b-310539ec92cf","Type":"ContainerStarted","Data":"b9df9258044f93717072afa42a189afb75ddf6bfaf3661872948cce66b544834"}
Jul 28 21:38:00.341888 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:00.341849    2565 generic.go:356] "Generic (PLEG): container finished" podID="1f252d24-58a1-49d5-be7b-310539ec92cf" containerID="ceaabb27116893d7c013416a0bed0110cb05ecf6265511ae851934658847dfe9" exitCode=0
Jul 28 21:38:00.342301 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:00.341899    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7298-140669728645680" event={"ID":"1f252d24-58a1-49d5-be7b-310539ec92cf","Type":"ContainerDied","Data":"ceaabb27116893d7c013416a0bed0110cb05ecf6265511ae851934658847dfe9"}
Jul 28 21:38:01.485122 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:01.485098    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728645680"
Jul 28 21:38:01.507089 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:01.507053    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7298-140669728645680_1f252d24-58a1-49d5-be7b-310539ec92cf/test-curl-7298-140669728645680/0.log"
Jul 28 21:38:01.529068 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:01.529038    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7298-140669728645680"]
Jul 28 21:38:01.532874 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:01.532851    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7298-140669728645680"]
Jul 28 21:38:01.577521 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:01.577490    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmh5j\" (UniqueName: \"kubernetes.io/projected/1f252d24-58a1-49d5-be7b-310539ec92cf-kube-api-access-fmh5j\") pod \"1f252d24-58a1-49d5-be7b-310539ec92cf\" (UID: \"1f252d24-58a1-49d5-be7b-310539ec92cf\") "
Jul 28 21:38:01.579571 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:01.579548    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f252d24-58a1-49d5-be7b-310539ec92cf-kube-api-access-fmh5j" (OuterVolumeSpecName: "kube-api-access-fmh5j") pod "1f252d24-58a1-49d5-be7b-310539ec92cf" (UID: "1f252d24-58a1-49d5-be7b-310539ec92cf"). InnerVolumeSpecName "kube-api-access-fmh5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:38:01.678782 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:01.678696    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmh5j\" (UniqueName: \"kubernetes.io/projected/1f252d24-58a1-49d5-be7b-310539ec92cf-kube-api-access-fmh5j\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:38:01.701657 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:01.701603    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7298-140669728426000"]
Jul 28 21:38:01.701992 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:01.701978    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f252d24-58a1-49d5-be7b-310539ec92cf" containerName="test-curl-7298-140669728645680"
Jul 28 21:38:01.702039 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:01.701993    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f252d24-58a1-49d5-be7b-310539ec92cf" containerName="test-curl-7298-140669728645680"
Jul 28 21:38:01.702073 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:01.702068    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f252d24-58a1-49d5-be7b-310539ec92cf" containerName="test-curl-7298-140669728645680"
Jul 28 21:38:01.705105 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:01.705088    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728426000"
Jul 28 21:38:01.709215 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:01.709188    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7298-140669728426000"]
Jul 28 21:38:01.780001 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:01.779969    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5fcc\" (UniqueName: \"kubernetes.io/projected/0a839d42-22a6-4b23-b337-eb588ae6f176-kube-api-access-v5fcc\") pod \"test-curl-7298-140669728426000\" (UID: \"0a839d42-22a6-4b23-b337-eb588ae6f176\") " pod="opendatahub/test-curl-7298-140669728426000"
Jul 28 21:38:01.881130 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:01.881091    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5fcc\" (UniqueName: \"kubernetes.io/projected/0a839d42-22a6-4b23-b337-eb588ae6f176-kube-api-access-v5fcc\") pod \"test-curl-7298-140669728426000\" (UID: \"0a839d42-22a6-4b23-b337-eb588ae6f176\") " pod="opendatahub/test-curl-7298-140669728426000"
Jul 28 21:38:01.894268 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:01.894239    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5fcc\" (UniqueName: \"kubernetes.io/projected/0a839d42-22a6-4b23-b337-eb588ae6f176-kube-api-access-v5fcc\") pod \"test-curl-7298-140669728426000\" (UID: \"0a839d42-22a6-4b23-b337-eb588ae6f176\") " pod="opendatahub/test-curl-7298-140669728426000"
Jul 28 21:38:02.015897 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:02.015860    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728426000"
Jul 28 21:38:02.150312 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:02.150289    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7298-140669728426000"]
Jul 28 21:38:02.152157 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:38:02.152129    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a839d42_22a6_4b23_b337_eb588ae6f176.slice/crio-db9c1abed025ddd217c087f0988d8d219ab91708a339328ad1e6bf4e0e054c10 WatchSource:0}: Error finding container db9c1abed025ddd217c087f0988d8d219ab91708a339328ad1e6bf4e0e054c10: Status 404 returned error can't find the container with id db9c1abed025ddd217c087f0988d8d219ab91708a339328ad1e6bf4e0e054c10
Jul 28 21:38:02.352705 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:02.352613    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728645680"
Jul 28 21:38:02.352705 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:02.352656    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9df9258044f93717072afa42a189afb75ddf6bfaf3661872948cce66b544834"
Jul 28 21:38:02.353801 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:02.353777    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7298-140669728426000" event={"ID":"0a839d42-22a6-4b23-b337-eb588ae6f176","Type":"ContainerStarted","Data":"db9c1abed025ddd217c087f0988d8d219ab91708a339328ad1e6bf4e0e054c10"}
Jul 28 21:38:02.363086 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:38:02.363053    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7298-140669728645680\" is forbidden: User \"system:node:ip-10-0-134-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-170.ec2.internal' and this object" podUID="1f252d24-58a1-49d5-be7b-310539ec92cf" pod="opendatahub/test-curl-7298-140669728645680"
Jul 28 21:38:03.331810 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:03.331778    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f252d24-58a1-49d5-be7b-310539ec92cf" path="/var/lib/kubelet/pods/1f252d24-58a1-49d5-be7b-310539ec92cf/volumes"
Jul 28 21:38:03.361101 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:03.361067    2565 generic.go:356] "Generic (PLEG): container finished" podID="0a839d42-22a6-4b23-b337-eb588ae6f176" containerID="6e2b7d1db39857d905ccbacc2a944cd38affcd4af89466c999a8616a636bff19" exitCode=0
Jul 28 21:38:03.361255 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:03.361162    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7298-140669728426000" event={"ID":"0a839d42-22a6-4b23-b337-eb588ae6f176","Type":"ContainerDied","Data":"6e2b7d1db39857d905ccbacc2a944cd38affcd4af89466c999a8616a636bff19"}
Jul 28 21:38:04.499732 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:04.499706    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728426000"
Jul 28 21:38:04.518383 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:04.518358    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7298-140669728426000_0a839d42-22a6-4b23-b337-eb588ae6f176/test-curl-7298-140669728426000/0.log"
Jul 28 21:38:04.538746 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:04.538711    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7298-140669728426000"]
Jul 28 21:38:04.544078 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:04.544055    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7298-140669728426000"]
Jul 28 21:38:04.605902 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:04.605876    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5fcc\" (UniqueName: \"kubernetes.io/projected/0a839d42-22a6-4b23-b337-eb588ae6f176-kube-api-access-v5fcc\") pod \"0a839d42-22a6-4b23-b337-eb588ae6f176\" (UID: \"0a839d42-22a6-4b23-b337-eb588ae6f176\") "
Jul 28 21:38:04.607903 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:04.607872    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a839d42-22a6-4b23-b337-eb588ae6f176-kube-api-access-v5fcc" (OuterVolumeSpecName: "kube-api-access-v5fcc") pod "0a839d42-22a6-4b23-b337-eb588ae6f176" (UID: "0a839d42-22a6-4b23-b337-eb588ae6f176"). InnerVolumeSpecName "kube-api-access-v5fcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:38:04.664143 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:04.664067    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7298-140669728426000"]
Jul 28 21:38:04.664445 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:04.664432    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a839d42-22a6-4b23-b337-eb588ae6f176" containerName="test-curl-7298-140669728426000"
Jul 28 21:38:04.664487 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:04.664447    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a839d42-22a6-4b23-b337-eb588ae6f176" containerName="test-curl-7298-140669728426000"
Jul 28 21:38:04.664537 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:04.664528    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a839d42-22a6-4b23-b337-eb588ae6f176" containerName="test-curl-7298-140669728426000"
Jul 28 21:38:04.667595 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:04.667580    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728426000"
Jul 28 21:38:04.674624 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:04.674597    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7298-140669728426000"]
Jul 28 21:38:04.706582 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:04.706551    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5fcc\" (UniqueName: \"kubernetes.io/projected/0a839d42-22a6-4b23-b337-eb588ae6f176-kube-api-access-v5fcc\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:38:04.807353 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:04.807320    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6znjf\" (UniqueName: \"kubernetes.io/projected/27c01459-9eb6-4590-aa5a-8be160384784-kube-api-access-6znjf\") pod \"test-curl-7298-140669728426000\" (UID: \"27c01459-9eb6-4590-aa5a-8be160384784\") " pod="opendatahub/test-curl-7298-140669728426000"
Jul 28 21:38:04.908078 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:04.908039    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6znjf\" (UniqueName: \"kubernetes.io/projected/27c01459-9eb6-4590-aa5a-8be160384784-kube-api-access-6znjf\") pod \"test-curl-7298-140669728426000\" (UID: \"27c01459-9eb6-4590-aa5a-8be160384784\") " pod="opendatahub/test-curl-7298-140669728426000"
Jul 28 21:38:04.917742 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:04.917676    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6znjf\" (UniqueName: \"kubernetes.io/projected/27c01459-9eb6-4590-aa5a-8be160384784-kube-api-access-6znjf\") pod \"test-curl-7298-140669728426000\" (UID: \"27c01459-9eb6-4590-aa5a-8be160384784\") " pod="opendatahub/test-curl-7298-140669728426000"
Jul 28 21:38:04.978567 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:04.978527    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728426000"
Jul 28 21:38:05.114170 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:05.114139    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7298-140669728426000"]
Jul 28 21:38:05.115999 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:38:05.115975    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27c01459_9eb6_4590_aa5a_8be160384784.slice/crio-f48935a4b3c7b137a78846ba88dae6dd6c0fa2a480a3384f82f6ff0959bfb70e WatchSource:0}: Error finding container f48935a4b3c7b137a78846ba88dae6dd6c0fa2a480a3384f82f6ff0959bfb70e: Status 404 returned error can't find the container with id f48935a4b3c7b137a78846ba88dae6dd6c0fa2a480a3384f82f6ff0959bfb70e
Jul 28 21:38:05.331869 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:05.331836    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a839d42-22a6-4b23-b337-eb588ae6f176" path="/var/lib/kubelet/pods/0a839d42-22a6-4b23-b337-eb588ae6f176/volumes"
Jul 28 21:38:05.373589 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:05.373561    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728426000"
Jul 28 21:38:05.373805 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:05.373600    2565 scope.go:117] "RemoveContainer" containerID="6e2b7d1db39857d905ccbacc2a944cd38affcd4af89466c999a8616a636bff19"
Jul 28 21:38:05.374978 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:05.374924    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7298-140669728426000" event={"ID":"27c01459-9eb6-4590-aa5a-8be160384784","Type":"ContainerStarted","Data":"f48935a4b3c7b137a78846ba88dae6dd6c0fa2a480a3384f82f6ff0959bfb70e"}
Jul 28 21:38:06.360537 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:06.360495    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:38:06.381657 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:06.381594    2565 generic.go:356] "Generic (PLEG): container finished" podID="27c01459-9eb6-4590-aa5a-8be160384784" containerID="178c288d8a79c4bc304e803e4090318f0836f02a561479a69381082e8fbd5bea" exitCode=0
Jul 28 21:38:06.381828 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:06.381685    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7298-140669728426000" event={"ID":"27c01459-9eb6-4590-aa5a-8be160384784","Type":"ContainerDied","Data":"178c288d8a79c4bc304e803e4090318f0836f02a561479a69381082e8fbd5bea"}
Jul 28 21:38:07.530995 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:07.530970    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728426000"
Jul 28 21:38:07.550754 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:07.550723    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7298-140669728426000_27c01459-9eb6-4590-aa5a-8be160384784/test-curl-7298-140669728426000/0.log"
Jul 28 21:38:07.573271 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:07.573230    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7298-140669728426000"]
Jul 28 21:38:07.576926 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:07.576902    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7298-140669728426000"]
Jul 28 21:38:07.630345 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:07.630307    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6znjf\" (UniqueName: \"kubernetes.io/projected/27c01459-9eb6-4590-aa5a-8be160384784-kube-api-access-6znjf\") pod \"27c01459-9eb6-4590-aa5a-8be160384784\" (UID: \"27c01459-9eb6-4590-aa5a-8be160384784\") "
Jul 28 21:38:07.632408 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:07.632376    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c01459-9eb6-4590-aa5a-8be160384784-kube-api-access-6znjf" (OuterVolumeSpecName: "kube-api-access-6znjf") pod "27c01459-9eb6-4590-aa5a-8be160384784" (UID: "27c01459-9eb6-4590-aa5a-8be160384784"). InnerVolumeSpecName "kube-api-access-6znjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:38:07.704044 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:07.704012    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7298-140669728427008"]
Jul 28 21:38:07.704397 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:07.704383    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27c01459-9eb6-4590-aa5a-8be160384784" containerName="test-curl-7298-140669728426000"
Jul 28 21:38:07.704447 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:07.704399    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="27c01459-9eb6-4590-aa5a-8be160384784" containerName="test-curl-7298-140669728426000"
Jul 28 21:38:07.704489 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:07.704470    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="27c01459-9eb6-4590-aa5a-8be160384784" containerName="test-curl-7298-140669728426000"
Jul 28 21:38:07.707465 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:07.707448    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728427008"
Jul 28 21:38:07.714479 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:07.714452    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7298-140669728427008"]
Jul 28 21:38:07.731474 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:07.731446    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6znjf\" (UniqueName: \"kubernetes.io/projected/27c01459-9eb6-4590-aa5a-8be160384784-kube-api-access-6znjf\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:38:07.831919 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:07.831878    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mj2m\" (UniqueName: \"kubernetes.io/projected/6209632c-d642-4eab-9c22-c35f45ceb339-kube-api-access-5mj2m\") pod \"test-curl-7298-140669728427008\" (UID: \"6209632c-d642-4eab-9c22-c35f45ceb339\") " pod="opendatahub/test-curl-7298-140669728427008"
Jul 28 21:38:07.932703 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:07.932608    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mj2m\" (UniqueName: \"kubernetes.io/projected/6209632c-d642-4eab-9c22-c35f45ceb339-kube-api-access-5mj2m\") pod \"test-curl-7298-140669728427008\" (UID: \"6209632c-d642-4eab-9c22-c35f45ceb339\") " pod="opendatahub/test-curl-7298-140669728427008"
Jul 28 21:38:07.940870 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:07.940839    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mj2m\" (UniqueName: \"kubernetes.io/projected/6209632c-d642-4eab-9c22-c35f45ceb339-kube-api-access-5mj2m\") pod \"test-curl-7298-140669728427008\" (UID: \"6209632c-d642-4eab-9c22-c35f45ceb339\") " pod="opendatahub/test-curl-7298-140669728427008"
Jul 28 21:38:08.017708 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:08.017672    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728427008"
Jul 28 21:38:08.170782 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:08.170755    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7298-140669728427008"]
Jul 28 21:38:08.172858 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:38:08.172823    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6209632c_d642_4eab_9c22_c35f45ceb339.slice/crio-af660972b772d5d59ae49e84fa088494f9ffc0a3b26f21da182fffc568d6c8d4 WatchSource:0}: Error finding container af660972b772d5d59ae49e84fa088494f9ffc0a3b26f21da182fffc568d6c8d4: Status 404 returned error can't find the container with id af660972b772d5d59ae49e84fa088494f9ffc0a3b26f21da182fffc568d6c8d4
Jul 28 21:38:08.395038 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:08.395009    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728426000"
Jul 28 21:38:08.395038 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:08.395035    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f48935a4b3c7b137a78846ba88dae6dd6c0fa2a480a3384f82f6ff0959bfb70e"
Jul 28 21:38:08.396315 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:08.396289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7298-140669728427008" event={"ID":"6209632c-d642-4eab-9c22-c35f45ceb339","Type":"ContainerStarted","Data":"af660972b772d5d59ae49e84fa088494f9ffc0a3b26f21da182fffc568d6c8d4"}
Jul 28 21:38:08.405441 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:38:08.405412    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7298-140669728426000\" is forbidden: User \"system:node:ip-10-0-134-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-170.ec2.internal' and this object" podUID="27c01459-9eb6-4590-aa5a-8be160384784" pod="opendatahub/test-curl-7298-140669728426000"
Jul 28 21:38:09.331491 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:09.331456    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27c01459-9eb6-4590-aa5a-8be160384784" path="/var/lib/kubelet/pods/27c01459-9eb6-4590-aa5a-8be160384784/volumes"
Jul 28 21:38:09.403421 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:09.403385    2565 generic.go:356] "Generic (PLEG): container finished" podID="6209632c-d642-4eab-9c22-c35f45ceb339" containerID="a4ce530337cbbf67c72a4e77bbb68b6f4e141b3a1acd1fcb7e363f68a43f1ff9" exitCode=0
Jul 28 21:38:09.403589 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:09.403462    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7298-140669728427008" event={"ID":"6209632c-d642-4eab-9c22-c35f45ceb339","Type":"ContainerDied","Data":"a4ce530337cbbf67c72a4e77bbb68b6f4e141b3a1acd1fcb7e363f68a43f1ff9"}
Jul 28 21:38:09.764450 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:09.764405    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9" podUID="86da8170-d97d-490b-92aa-15e27f6aa7ae" containerName="main" containerID="cri-o://74941fcdaabbd23ad5f93662b0e2f0408ebc3011ecf5d2dfdb7c798547c5bac6" gracePeriod=45
Jul 28 21:38:10.027292 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.025234    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:38:10.153097 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.153063    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-home\") pod \"86da8170-d97d-490b-92aa-15e27f6aa7ae\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") "
Jul 28 21:38:10.153314 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.153117    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-tmp-dir\") pod \"86da8170-d97d-490b-92aa-15e27f6aa7ae\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") "
Jul 28 21:38:10.153314 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.153150    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-dshm\") pod \"86da8170-d97d-490b-92aa-15e27f6aa7ae\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") "
Jul 28 21:38:10.153314 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.153189    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxv54\" (UniqueName: \"kubernetes.io/projected/86da8170-d97d-490b-92aa-15e27f6aa7ae-kube-api-access-dxv54\") pod \"86da8170-d97d-490b-92aa-15e27f6aa7ae\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") "
Jul 28 21:38:10.153314 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.153217    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86da8170-d97d-490b-92aa-15e27f6aa7ae-tls-certs\") pod \"86da8170-d97d-490b-92aa-15e27f6aa7ae\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") "
Jul 28 21:38:10.153314 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.153267    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-model-cache\") pod \"86da8170-d97d-490b-92aa-15e27f6aa7ae\" (UID: \"86da8170-d97d-490b-92aa-15e27f6aa7ae\") "
Jul 28 21:38:10.153565 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.153407    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-home" (OuterVolumeSpecName: "home") pod "86da8170-d97d-490b-92aa-15e27f6aa7ae" (UID: "86da8170-d97d-490b-92aa-15e27f6aa7ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:38:10.153565 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.153483    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86da8170-d97d-490b-92aa-15e27f6aa7ae" (UID: "86da8170-d97d-490b-92aa-15e27f6aa7ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:38:10.153716 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.153594    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "86da8170-d97d-490b-92aa-15e27f6aa7ae" (UID: "86da8170-d97d-490b-92aa-15e27f6aa7ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:38:10.155324 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.155306    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-dshm" (OuterVolumeSpecName: "dshm") pod "86da8170-d97d-490b-92aa-15e27f6aa7ae" (UID: "86da8170-d97d-490b-92aa-15e27f6aa7ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 21:38:10.155394 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.155328    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86da8170-d97d-490b-92aa-15e27f6aa7ae-kube-api-access-dxv54" (OuterVolumeSpecName: "kube-api-access-dxv54") pod "86da8170-d97d-490b-92aa-15e27f6aa7ae" (UID: "86da8170-d97d-490b-92aa-15e27f6aa7ae"). InnerVolumeSpecName "kube-api-access-dxv54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:38:10.155433 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.155407    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86da8170-d97d-490b-92aa-15e27f6aa7ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86da8170-d97d-490b-92aa-15e27f6aa7ae" (UID: "86da8170-d97d-490b-92aa-15e27f6aa7ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:38:10.254332 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.254295    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-model-cache\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:38:10.254332 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.254325    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-home\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:38:10.254332 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.254338    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-tmp-dir\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:38:10.254594 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.254349    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86da8170-d97d-490b-92aa-15e27f6aa7ae-dshm\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:38:10.254594 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.254361    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxv54\" (UniqueName: \"kubernetes.io/projected/86da8170-d97d-490b-92aa-15e27f6aa7ae-kube-api-access-dxv54\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:38:10.254594 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.254371    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86da8170-d97d-490b-92aa-15e27f6aa7ae-tls-certs\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:38:10.411788 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.411705    2565 generic.go:356] "Generic (PLEG): container finished" podID="86da8170-d97d-490b-92aa-15e27f6aa7ae" containerID="74941fcdaabbd23ad5f93662b0e2f0408ebc3011ecf5d2dfdb7c798547c5bac6" exitCode=0
Jul 28 21:38:10.411788 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.411745    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"
Jul 28 21:38:10.412380 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.411792    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9" event={"ID":"86da8170-d97d-490b-92aa-15e27f6aa7ae","Type":"ContainerDied","Data":"74941fcdaabbd23ad5f93662b0e2f0408ebc3011ecf5d2dfdb7c798547c5bac6"}
Jul 28 21:38:10.412380 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.411836    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9" event={"ID":"86da8170-d97d-490b-92aa-15e27f6aa7ae","Type":"ContainerDied","Data":"0fc59cd8d8ac8c4ab5f145a0902382d01faf68273832a313f0d3fa21b1b72eb6"}
Jul 28 21:38:10.412380 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.411859    2565 scope.go:117] "RemoveContainer" containerID="74941fcdaabbd23ad5f93662b0e2f0408ebc3011ecf5d2dfdb7c798547c5bac6"
Jul 28 21:38:10.421574 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.421548    2565 scope.go:117] "RemoveContainer" containerID="74941fcdaabbd23ad5f93662b0e2f0408ebc3011ecf5d2dfdb7c798547c5bac6"
Jul 28 21:38:10.421902 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:38:10.421882    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74941fcdaabbd23ad5f93662b0e2f0408ebc3011ecf5d2dfdb7c798547c5bac6\": container with ID starting with 74941fcdaabbd23ad5f93662b0e2f0408ebc3011ecf5d2dfdb7c798547c5bac6 not found: ID does not exist" containerID="74941fcdaabbd23ad5f93662b0e2f0408ebc3011ecf5d2dfdb7c798547c5bac6"
Jul 28 21:38:10.421981 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.421912    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74941fcdaabbd23ad5f93662b0e2f0408ebc3011ecf5d2dfdb7c798547c5bac6"} err="failed to get container status \"74941fcdaabbd23ad5f93662b0e2f0408ebc3011ecf5d2dfdb7c798547c5bac6\": rpc error: code = NotFound desc = could not find container \"74941fcdaabbd23ad5f93662b0e2f0408ebc3011ecf5d2dfdb7c798547c5bac6\": container with ID starting with 74941fcdaabbd23ad5f93662b0e2f0408ebc3011ecf5d2dfdb7c798547c5bac6 not found: ID does not exist"
Jul 28 21:38:10.434513 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.434488    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"]
Jul 28 21:38:10.436671 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.436630    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-2c3f70/rate-test-model-2c3f70-kserve-588f9785f9-b58k9"]
Jul 28 21:38:10.539145 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.539120    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728427008"
Jul 28 21:38:10.559069 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.559038    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7298-140669728427008_6209632c-d642-4eab-9c22-c35f45ceb339/test-curl-7298-140669728427008/0.log"
Jul 28 21:38:10.581598 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.581562    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7298-140669728427008"]
Jul 28 21:38:10.588660 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.588591    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7298-140669728427008"]
Jul 28 21:38:10.657602 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.657565    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mj2m\" (UniqueName: \"kubernetes.io/projected/6209632c-d642-4eab-9c22-c35f45ceb339-kube-api-access-5mj2m\") pod \"6209632c-d642-4eab-9c22-c35f45ceb339\" (UID: \"6209632c-d642-4eab-9c22-c35f45ceb339\") "
Jul 28 21:38:10.659680 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.659653    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6209632c-d642-4eab-9c22-c35f45ceb339-kube-api-access-5mj2m" (OuterVolumeSpecName: "kube-api-access-5mj2m") pod "6209632c-d642-4eab-9c22-c35f45ceb339" (UID: "6209632c-d642-4eab-9c22-c35f45ceb339"). InnerVolumeSpecName "kube-api-access-5mj2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:38:10.706915 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.706881    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7298-140669728427008"]
Jul 28 21:38:10.707271 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.707258    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86da8170-d97d-490b-92aa-15e27f6aa7ae" containerName="main"
Jul 28 21:38:10.707313 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.707272    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="86da8170-d97d-490b-92aa-15e27f6aa7ae" containerName="main"
Jul 28 21:38:10.707313 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.707289    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6209632c-d642-4eab-9c22-c35f45ceb339" containerName="test-curl-7298-140669728427008"
Jul 28 21:38:10.707313 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.707296    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6209632c-d642-4eab-9c22-c35f45ceb339" containerName="test-curl-7298-140669728427008"
Jul 28 21:38:10.707415 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.707364    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="86da8170-d97d-490b-92aa-15e27f6aa7ae" containerName="main"
Jul 28 21:38:10.707415 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.707374    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6209632c-d642-4eab-9c22-c35f45ceb339" containerName="test-curl-7298-140669728427008"
Jul 28 21:38:10.710475 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.710458    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728427008"
Jul 28 21:38:10.717108 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.717083    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7298-140669728427008"]
Jul 28 21:38:10.758557 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.758523    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mj2m\" (UniqueName: \"kubernetes.io/projected/6209632c-d642-4eab-9c22-c35f45ceb339-kube-api-access-5mj2m\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:38:10.858952 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.858916    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp98c\" (UniqueName: \"kubernetes.io/projected/a303e1d5-6d73-4b5f-995f-9244319d8c70-kube-api-access-wp98c\") pod \"test-curl-7298-140669728427008\" (UID: \"a303e1d5-6d73-4b5f-995f-9244319d8c70\") " pod="opendatahub/test-curl-7298-140669728427008"
Jul 28 21:38:10.960243 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.960148    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp98c\" (UniqueName: \"kubernetes.io/projected/a303e1d5-6d73-4b5f-995f-9244319d8c70-kube-api-access-wp98c\") pod \"test-curl-7298-140669728427008\" (UID: \"a303e1d5-6d73-4b5f-995f-9244319d8c70\") " pod="opendatahub/test-curl-7298-140669728427008"
Jul 28 21:38:10.970037 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:10.970014    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp98c\" (UniqueName: \"kubernetes.io/projected/a303e1d5-6d73-4b5f-995f-9244319d8c70-kube-api-access-wp98c\") pod \"test-curl-7298-140669728427008\" (UID: \"a303e1d5-6d73-4b5f-995f-9244319d8c70\") " pod="opendatahub/test-curl-7298-140669728427008"
Jul 28 21:38:11.020756 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:11.020718    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728427008"
Jul 28 21:38:11.154241 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:11.154211    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7298-140669728427008"]
Jul 28 21:38:11.155821 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:38:11.155787    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda303e1d5_6d73_4b5f_995f_9244319d8c70.slice/crio-aa75423ecd1bbd0de02213dd7ab73ab8038e199761d43d38c775cfea990b6309 WatchSource:0}: Error finding container aa75423ecd1bbd0de02213dd7ab73ab8038e199761d43d38c775cfea990b6309: Status 404 returned error can't find the container with id aa75423ecd1bbd0de02213dd7ab73ab8038e199761d43d38c775cfea990b6309
Jul 28 21:38:11.332552 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:11.332459    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6209632c-d642-4eab-9c22-c35f45ceb339" path="/var/lib/kubelet/pods/6209632c-d642-4eab-9c22-c35f45ceb339/volumes"
Jul 28 21:38:11.332992 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:11.332970    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86da8170-d97d-490b-92aa-15e27f6aa7ae" path="/var/lib/kubelet/pods/86da8170-d97d-490b-92aa-15e27f6aa7ae/volumes"
Jul 28 21:38:11.419798 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:11.419763    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7298-140669728427008" event={"ID":"a303e1d5-6d73-4b5f-995f-9244319d8c70","Type":"ContainerStarted","Data":"aa75423ecd1bbd0de02213dd7ab73ab8038e199761d43d38c775cfea990b6309"}
Jul 28 21:38:11.420995 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:11.420974    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728427008"
Jul 28 21:38:11.420995 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:11.420986    2565 scope.go:117] "RemoveContainer" containerID="a4ce530337cbbf67c72a4e77bbb68b6f4e141b3a1acd1fcb7e363f68a43f1ff9"
Jul 28 21:38:12.429024 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:12.428993    2565 generic.go:356] "Generic (PLEG): container finished" podID="a303e1d5-6d73-4b5f-995f-9244319d8c70" containerID="683e12afc224038fb9728133602418d726782348a9fc27ecc124eecbd081cac2" exitCode=0
Jul 28 21:38:12.429479 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:12.429089    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7298-140669728427008" event={"ID":"a303e1d5-6d73-4b5f-995f-9244319d8c70","Type":"ContainerDied","Data":"683e12afc224038fb9728133602418d726782348a9fc27ecc124eecbd081cac2"}
Jul 28 21:38:13.565728 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:13.565705    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728427008"
Jul 28 21:38:13.585544 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:13.585492    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7298-140669728427008_a303e1d5-6d73-4b5f-995f-9244319d8c70/test-curl-7298-140669728427008/0.log"
Jul 28 21:38:13.609480 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:13.609448    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7298-140669728427008"]
Jul 28 21:38:13.611659 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:13.611611    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7298-140669728427008"]
Jul 28 21:38:13.686015 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:13.685917    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp98c\" (UniqueName: \"kubernetes.io/projected/a303e1d5-6d73-4b5f-995f-9244319d8c70-kube-api-access-wp98c\") pod \"a303e1d5-6d73-4b5f-995f-9244319d8c70\" (UID: \"a303e1d5-6d73-4b5f-995f-9244319d8c70\") "
Jul 28 21:38:13.687979 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:13.687951    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a303e1d5-6d73-4b5f-995f-9244319d8c70-kube-api-access-wp98c" (OuterVolumeSpecName: "kube-api-access-wp98c") pod "a303e1d5-6d73-4b5f-995f-9244319d8c70" (UID: "a303e1d5-6d73-4b5f-995f-9244319d8c70"). InnerVolumeSpecName "kube-api-access-wp98c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:38:13.787454 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:13.787413    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp98c\" (UniqueName: \"kubernetes.io/projected/a303e1d5-6d73-4b5f-995f-9244319d8c70-kube-api-access-wp98c\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:38:14.444310 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:14.444283    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa75423ecd1bbd0de02213dd7ab73ab8038e199761d43d38c775cfea990b6309"
Jul 28 21:38:14.444310 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:14.444285    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7298-140669728427008"
Jul 28 21:38:14.453709 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:38:14.453678    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7298-140669728427008\" is forbidden: User \"system:node:ip-10-0-134-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-170.ec2.internal' and this object" podUID="a303e1d5-6d73-4b5f-995f-9244319d8c70" pod="opendatahub/test-curl-7298-140669728427008"
Jul 28 21:38:15.330251 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:38:15.330218    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7298-140669728427008\" is forbidden: User \"system:node:ip-10-0-134-170.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-170.ec2.internal' and this object" podUID="a303e1d5-6d73-4b5f-995f-9244319d8c70" pod="opendatahub/test-curl-7298-140669728427008"
Jul 28 21:38:15.331879 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:15.331858    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a303e1d5-6d73-4b5f-995f-9244319d8c70" path="/var/lib/kubelet/pods/a303e1d5-6d73-4b5f-995f-9244319d8c70/volumes"
Jul 28 21:38:17.714555 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:17.714477    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"]
Jul 28 21:38:17.715043 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:17.715026    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a303e1d5-6d73-4b5f-995f-9244319d8c70" containerName="test-curl-7298-140669728427008"
Jul 28 21:38:17.715088 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:17.715045    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a303e1d5-6d73-4b5f-995f-9244319d8c70" containerName="test-curl-7298-140669728427008"
Jul 28 21:38:17.715127 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:17.715119    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a303e1d5-6d73-4b5f-995f-9244319d8c70" containerName="test-curl-7298-140669728427008"
Jul 28 21:38:17.718357 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:17.718338    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"
Jul 28 21:38:17.720194 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:17.720173    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-cf1484-serving-cert\""
Jul 28 21:38:17.724460 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:17.724436    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7115d06e-9be9-4b77-a390-87beaa33cb92-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz\" (UID: \"7115d06e-9be9-4b77-a390-87beaa33cb92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"
Jul 28 21:38:17.724574 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:17.724504    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dbnf\" (UniqueName: \"kubernetes.io/projected/7115d06e-9be9-4b77-a390-87beaa33cb92-kube-api-access-8dbnf\") pod \"maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz\" (UID: \"7115d06e-9be9-4b77-a390-87beaa33cb92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"
Jul 28 21:38:17.724574 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:17.724546    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7115d06e-9be9-4b77-a390-87beaa33cb92-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz\" (UID: \"7115d06e-9be9-4b77-a390-87beaa33cb92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"
Jul 28 21:38:17.726771 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:17.726748    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"]
Jul 28 21:38:17.825260 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:17.825216    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7115d06e-9be9-4b77-a390-87beaa33cb92-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz\" (UID: \"7115d06e-9be9-4b77-a390-87beaa33cb92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"
Jul 28 21:38:17.825479 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:17.825277    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dbnf\" (UniqueName: \"kubernetes.io/projected/7115d06e-9be9-4b77-a390-87beaa33cb92-kube-api-access-8dbnf\") pod \"maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz\" (UID: \"7115d06e-9be9-4b77-a390-87beaa33cb92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"
Jul 28 21:38:17.825479 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:17.825303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7115d06e-9be9-4b77-a390-87beaa33cb92-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz\" (UID: \"7115d06e-9be9-4b77-a390-87beaa33cb92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"
Jul 28 21:38:17.825479 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:38:17.825366    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-serving-cert: secret "maas-api-e2e-ipp-a-cf1484-serving-cert" not found
Jul 28 21:38:17.825479 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:38:17.825475    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7115d06e-9be9-4b77-a390-87beaa33cb92-maas-api-tls podName:7115d06e-9be9-4b77-a390-87beaa33cb92 nodeName:}" failed. No retries permitted until 2026-07-28 21:38:18.325453801 +0000 UTC m=+3053.549796779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7115d06e-9be9-4b77-a390-87beaa33cb92-maas-api-tls") pod "maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz" (UID: "7115d06e-9be9-4b77-a390-87beaa33cb92") : secret "maas-api-e2e-ipp-a-cf1484-serving-cert" not found
Jul 28 21:38:17.825945 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:17.825926    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7115d06e-9be9-4b77-a390-87beaa33cb92-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz\" (UID: \"7115d06e-9be9-4b77-a390-87beaa33cb92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"
Jul 28 21:38:17.843205 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:17.843175    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dbnf\" (UniqueName: \"kubernetes.io/projected/7115d06e-9be9-4b77-a390-87beaa33cb92-kube-api-access-8dbnf\") pod \"maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz\" (UID: \"7115d06e-9be9-4b77-a390-87beaa33cb92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"
Jul 28 21:38:18.329013 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:18.328973    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7115d06e-9be9-4b77-a390-87beaa33cb92-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz\" (UID: \"7115d06e-9be9-4b77-a390-87beaa33cb92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"
Jul 28 21:38:18.331407 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:18.331389    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7115d06e-9be9-4b77-a390-87beaa33cb92-maas-api-tls\") pod \"maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz\" (UID: \"7115d06e-9be9-4b77-a390-87beaa33cb92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"
Jul 28 21:38:18.553376 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:18.553339    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl"]
Jul 28 21:38:18.557108 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:18.557092    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl"
Jul 28 21:38:18.559037 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:18.559016    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-cf1484-dockercfg-8gdqp\""
Jul 28 21:38:18.559150 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:18.559020    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-cf1484\""
Jul 28 21:38:18.564927 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:18.564903    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl"]
Jul 28 21:38:18.629998 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:18.629916    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"
Jul 28 21:38:18.630255 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:18.630232    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7720ded-b3d7-4810-852c-5d86c5f01893-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl\" (UID: \"d7720ded-b3d7-4810-852c-5d86c5f01893\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl"
Jul 28 21:38:18.630356 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:18.630326    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcdqv\" (UniqueName: \"kubernetes.io/projected/d7720ded-b3d7-4810-852c-5d86c5f01893-kube-api-access-jcdqv\") pod \"payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl\" (UID: \"d7720ded-b3d7-4810-852c-5d86c5f01893\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl"
Jul 28 21:38:18.731177 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:18.731144    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcdqv\" (UniqueName: \"kubernetes.io/projected/d7720ded-b3d7-4810-852c-5d86c5f01893-kube-api-access-jcdqv\") pod \"payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl\" (UID: \"d7720ded-b3d7-4810-852c-5d86c5f01893\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl"
Jul 28 21:38:18.731618 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:18.731197    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7720ded-b3d7-4810-852c-5d86c5f01893-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl\" (UID: \"d7720ded-b3d7-4810-852c-5d86c5f01893\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl"
Jul 28 21:38:18.731959 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:18.731938    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7720ded-b3d7-4810-852c-5d86c5f01893-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl\" (UID: \"d7720ded-b3d7-4810-852c-5d86c5f01893\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl"
Jul 28 21:38:18.739093 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:18.739062    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcdqv\" (UniqueName: \"kubernetes.io/projected/d7720ded-b3d7-4810-852c-5d86c5f01893-kube-api-access-jcdqv\") pod \"payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl\" (UID: \"d7720ded-b3d7-4810-852c-5d86c5f01893\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl"
Jul 28 21:38:18.765766 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:18.765738    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"]
Jul 28 21:38:18.767341 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:38:18.767313    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7115d06e_9be9_4b77_a390_87beaa33cb92.slice/crio-1d496c6fb31bdfefe7d165723f01c72dfac78d30fc910bdbdf9d5308f68c7b0c WatchSource:0}: Error finding container 1d496c6fb31bdfefe7d165723f01c72dfac78d30fc910bdbdf9d5308f68c7b0c: Status 404 returned error can't find the container with id 1d496c6fb31bdfefe7d165723f01c72dfac78d30fc910bdbdf9d5308f68c7b0c
Jul 28 21:38:18.868478 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:18.868442    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl"
Jul 28 21:38:19.003541 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:19.003516    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl"]
Jul 28 21:38:19.004461 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:38:19.004430    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7720ded_b3d7_4810_852c_5d86c5f01893.slice/crio-04327cb5b2588b84c8ad6614c937c26705ec6163f2d0abfde3e20874fd589a0f WatchSource:0}: Error finding container 04327cb5b2588b84c8ad6614c937c26705ec6163f2d0abfde3e20874fd589a0f: Status 404 returned error can't find the container with id 04327cb5b2588b84c8ad6614c937c26705ec6163f2d0abfde3e20874fd589a0f
Jul 28 21:38:19.476589 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:19.476530    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz" event={"ID":"7115d06e-9be9-4b77-a390-87beaa33cb92","Type":"ContainerStarted","Data":"0a48e71d226d1196fed553154f04815363a6d68825735bd094d48eaf072ce14b"}
Jul 28 21:38:19.476784 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:19.476595    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz" event={"ID":"7115d06e-9be9-4b77-a390-87beaa33cb92","Type":"ContainerStarted","Data":"1d496c6fb31bdfefe7d165723f01c72dfac78d30fc910bdbdf9d5308f68c7b0c"}
Jul 28 21:38:19.476784 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:19.476758    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"
Jul 28 21:38:19.478067 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:19.478044    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl" event={"ID":"d7720ded-b3d7-4810-852c-5d86c5f01893","Type":"ContainerStarted","Data":"48085b7dcde9f6e9782161d269794f4694ede0a89ed51e3d3aff85eb6f67dca5"}
Jul 28 21:38:19.478155 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:19.478090    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl" event={"ID":"d7720ded-b3d7-4810-852c-5d86c5f01893","Type":"ContainerStarted","Data":"04327cb5b2588b84c8ad6614c937c26705ec6163f2d0abfde3e20874fd589a0f"}
Jul 28 21:38:19.478155 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:19.478121    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl"
Jul 28 21:38:19.495401 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:19.495357    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz" podStartSLOduration=2.015371136 podStartE2EDuration="2.495343346s" totalImagesPullingTime="479.97221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:38:19.49247382 +0000 UTC m=+3054.716816806" watchObservedRunningTime="2026-07-28 21:38:19.495343346 +0000 UTC m=+3054.719686331"
Jul 28 21:38:19.508374 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:19.508285    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl" podStartSLOduration=1.5082652749999998 podStartE2EDuration="1.508265275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:38:19.506253559 +0000 UTC m=+3054.730596545" watchObservedRunningTime="2026-07-28 21:38:19.508265275 +0000 UTC m=+3054.732608262"
Jul 28 21:38:25.490290 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:25.490262    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"
Jul 28 21:38:25.857908 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:25.857818    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:38:27.174816 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:27.174776    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bgdnc"]
Jul 28 21:38:27.181001 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:27.180978    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bgdnc"
Jul 28 21:38:27.183624 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:27.183595    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-jvb4f\""
Jul 28 21:38:27.186455 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:27.186425    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bgdnc"]
Jul 28 21:38:27.205619 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:27.205593    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqzzx\" (UniqueName: \"kubernetes.io/projected/11956a62-79fb-4156-b211-5321f68b76b7-kube-api-access-gqzzx\") pod \"kuadrant-operator-catalog-bgdnc\" (UID: \"11956a62-79fb-4156-b211-5321f68b76b7\") " pod="kuadrant-system/kuadrant-operator-catalog-bgdnc"
Jul 28 21:38:27.306974 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:27.306929    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqzzx\" (UniqueName: \"kubernetes.io/projected/11956a62-79fb-4156-b211-5321f68b76b7-kube-api-access-gqzzx\") pod \"kuadrant-operator-catalog-bgdnc\" (UID: \"11956a62-79fb-4156-b211-5321f68b76b7\") " pod="kuadrant-system/kuadrant-operator-catalog-bgdnc"
Jul 28 21:38:27.314714 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:27.314679    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqzzx\" (UniqueName: \"kubernetes.io/projected/11956a62-79fb-4156-b211-5321f68b76b7-kube-api-access-gqzzx\") pod \"kuadrant-operator-catalog-bgdnc\" (UID: \"11956a62-79fb-4156-b211-5321f68b76b7\") " pod="kuadrant-system/kuadrant-operator-catalog-bgdnc"
Jul 28 21:38:27.492033 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:27.492010    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bgdnc"
Jul 28 21:38:27.634882 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:27.634858    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bgdnc"]
Jul 28 21:38:27.637216 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:38:27.637187    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11956a62_79fb_4156_b211_5321f68b76b7.slice/crio-cecd6af855a3240fc25d2a1f80c4e4f85a82c47a928d4d878af027405529b39f WatchSource:0}: Error finding container cecd6af855a3240fc25d2a1f80c4e4f85a82c47a928d4d878af027405529b39f: Status 404 returned error can't find the container with id cecd6af855a3240fc25d2a1f80c4e4f85a82c47a928d4d878af027405529b39f
Jul 28 21:38:28.535133 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:28.535090    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bgdnc" event={"ID":"11956a62-79fb-4156-b211-5321f68b76b7","Type":"ContainerStarted","Data":"cecd6af855a3240fc25d2a1f80c4e4f85a82c47a928d4d878af027405529b39f"}
Jul 28 21:38:30.486094 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:30.486063    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl"
Jul 28 21:38:30.547847 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:30.547801    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bgdnc" event={"ID":"11956a62-79fb-4156-b211-5321f68b76b7","Type":"ContainerStarted","Data":"bde41f0241679ee0662f35a20142298c07e4556e06a403b42916c31a5321eb97"}
Jul 28 21:38:30.564081 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:30.564031    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-bgdnc" podStartSLOduration=1.630293252 podStartE2EDuration="3.564014971s" totalImagesPullingTime="1.933721719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:38:30.562540763 +0000 UTC m=+3065.786883750" watchObservedRunningTime="2026-07-28 21:38:30.564014971 +0000 UTC m=+3065.788357957"
Jul 28 21:38:37.492986 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:37.492945    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-bgdnc"
Jul 28 21:38:37.493504 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:37.493020    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-bgdnc"
Jul 28 21:38:37.515062 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:37.515033    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-bgdnc"
Jul 28 21:38:37.608879 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:37.608848    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-bgdnc"
Jul 28 21:38:37.741232 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:37.741196    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bgdnc"]
Jul 28 21:38:39.597716 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:39.597674    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-bgdnc" podUID="11956a62-79fb-4156-b211-5321f68b76b7" containerName="registry-server" containerID="cri-o://bde41f0241679ee0662f35a20142298c07e4556e06a403b42916c31a5321eb97" gracePeriod=2
Jul 28 21:38:39.858159 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:39.858100    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bgdnc"
Jul 28 21:38:39.923534 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:39.923490    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqzzx\" (UniqueName: \"kubernetes.io/projected/11956a62-79fb-4156-b211-5321f68b76b7-kube-api-access-gqzzx\") pod \"11956a62-79fb-4156-b211-5321f68b76b7\" (UID: \"11956a62-79fb-4156-b211-5321f68b76b7\") "
Jul 28 21:38:39.925539 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:39.925508    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11956a62-79fb-4156-b211-5321f68b76b7-kube-api-access-gqzzx" (OuterVolumeSpecName: "kube-api-access-gqzzx") pod "11956a62-79fb-4156-b211-5321f68b76b7" (UID: "11956a62-79fb-4156-b211-5321f68b76b7"). InnerVolumeSpecName "kube-api-access-gqzzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:38:40.025161 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:40.025109    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqzzx\" (UniqueName: \"kubernetes.io/projected/11956a62-79fb-4156-b211-5321f68b76b7-kube-api-access-gqzzx\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:38:40.605145 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:40.605109    2565 generic.go:356] "Generic (PLEG): container finished" podID="11956a62-79fb-4156-b211-5321f68b76b7" containerID="bde41f0241679ee0662f35a20142298c07e4556e06a403b42916c31a5321eb97" exitCode=0
Jul 28 21:38:40.605623 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:40.605160    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bgdnc"
Jul 28 21:38:40.605623 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:40.605196    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bgdnc" event={"ID":"11956a62-79fb-4156-b211-5321f68b76b7","Type":"ContainerDied","Data":"bde41f0241679ee0662f35a20142298c07e4556e06a403b42916c31a5321eb97"}
Jul 28 21:38:40.605623 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:40.605251    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bgdnc" event={"ID":"11956a62-79fb-4156-b211-5321f68b76b7","Type":"ContainerDied","Data":"cecd6af855a3240fc25d2a1f80c4e4f85a82c47a928d4d878af027405529b39f"}
Jul 28 21:38:40.605623 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:40.605280    2565 scope.go:117] "RemoveContainer" containerID="bde41f0241679ee0662f35a20142298c07e4556e06a403b42916c31a5321eb97"
Jul 28 21:38:40.615989 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:40.615974    2565 scope.go:117] "RemoveContainer" containerID="bde41f0241679ee0662f35a20142298c07e4556e06a403b42916c31a5321eb97"
Jul 28 21:38:40.616226 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:38:40.616205    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde41f0241679ee0662f35a20142298c07e4556e06a403b42916c31a5321eb97\": container with ID starting with bde41f0241679ee0662f35a20142298c07e4556e06a403b42916c31a5321eb97 not found: ID does not exist" containerID="bde41f0241679ee0662f35a20142298c07e4556e06a403b42916c31a5321eb97"
Jul 28 21:38:40.616292 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:40.616234    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde41f0241679ee0662f35a20142298c07e4556e06a403b42916c31a5321eb97"} err="failed to get container status \"bde41f0241679ee0662f35a20142298c07e4556e06a403b42916c31a5321eb97\": rpc error: code = NotFound desc = could not find container \"bde41f0241679ee0662f35a20142298c07e4556e06a403b42916c31a5321eb97\": container with ID starting with bde41f0241679ee0662f35a20142298c07e4556e06a403b42916c31a5321eb97 not found: ID does not exist"
Jul 28 21:38:40.627126 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:40.627104    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bgdnc"]
Jul 28 21:38:40.630679 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:40.630659    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bgdnc"]
Jul 28 21:38:41.332223 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:41.332193    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11956a62-79fb-4156-b211-5321f68b76b7" path="/var/lib/kubelet/pods/11956a62-79fb-4156-b211-5321f68b76b7/volumes"
Jul 28 21:38:46.260504 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:46.260469    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"]
Jul 28 21:38:46.261059 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:46.261039    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11956a62-79fb-4156-b211-5321f68b76b7" containerName="registry-server"
Jul 28 21:38:46.261138 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:46.261062    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="11956a62-79fb-4156-b211-5321f68b76b7" containerName="registry-server"
Jul 28 21:38:46.261198 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:46.261181    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="11956a62-79fb-4156-b211-5321f68b76b7" containerName="registry-server"
Jul 28 21:38:46.265485 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:46.265465    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"
Jul 28 21:38:46.267332 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:46.267306    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d69596-serving-cert\""
Jul 28 21:38:46.276704 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:46.276672    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"]
Jul 28 21:38:46.383140 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:46.383100    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm\" (UID: \"11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"
Jul 28 21:38:46.383322 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:46.383151    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flt6k\" (UniqueName: \"kubernetes.io/projected/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-kube-api-access-flt6k\") pod \"maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm\" (UID: \"11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"
Jul 28 21:38:46.383322 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:46.383252    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm\" (UID: \"11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"
Jul 28 21:38:46.484490 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:46.484454    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm\" (UID: \"11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"
Jul 28 21:38:46.484686 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:46.484499    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flt6k\" (UniqueName: \"kubernetes.io/projected/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-kube-api-access-flt6k\") pod \"maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm\" (UID: \"11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"
Jul 28 21:38:46.484686 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:46.484546    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm\" (UID: \"11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"
Jul 28 21:38:46.484777 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:38:46.484739    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-serving-cert: secret "maas-api-e2e-ipp-b-d69596-serving-cert" not found
Jul 28 21:38:46.484825 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:38:46.484813    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-maas-api-tls podName:11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c nodeName:}" failed. No retries permitted until 2026-07-28 21:38:46.984790815 +0000 UTC m=+3082.209133782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-maas-api-tls") pod "maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm" (UID: "11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c") : secret "maas-api-e2e-ipp-b-d69596-serving-cert" not found
Jul 28 21:38:46.485237 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:46.485216    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm\" (UID: \"11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"
Jul 28 21:38:46.492707 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:46.492688    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flt6k\" (UniqueName: \"kubernetes.io/projected/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-kube-api-access-flt6k\") pod \"maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm\" (UID: \"11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"
Jul 28 21:38:46.988522 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:46.988442    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm\" (UID: \"11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"
Jul 28 21:38:46.990777 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:46.990758    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm\" (UID: \"11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"
Jul 28 21:38:47.063377 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:47.063332    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x"]
Jul 28 21:38:47.067024 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:47.066999    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x"
Jul 28 21:38:47.069320 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:47.069294    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d69596-dockercfg-52nwz\""
Jul 28 21:38:47.069451 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:47.069387    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d69596\""
Jul 28 21:38:47.079599 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:47.079573    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x"]
Jul 28 21:38:47.177798 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:47.177768    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"
Jul 28 21:38:47.190940 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:47.190915    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rztl9\" (UniqueName: \"kubernetes.io/projected/42444876-b7b4-44b7-b051-d70b44d906a3-kube-api-access-rztl9\") pod \"payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x\" (UID: \"42444876-b7b4-44b7-b051-d70b44d906a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x"
Jul 28 21:38:47.191035 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:47.190968    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42444876-b7b4-44b7-b051-d70b44d906a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x\" (UID: \"42444876-b7b4-44b7-b051-d70b44d906a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x"
Jul 28 21:38:47.292161 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:47.292122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rztl9\" (UniqueName: \"kubernetes.io/projected/42444876-b7b4-44b7-b051-d70b44d906a3-kube-api-access-rztl9\") pod \"payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x\" (UID: \"42444876-b7b4-44b7-b051-d70b44d906a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x"
Jul 28 21:38:47.292873 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:47.292208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42444876-b7b4-44b7-b051-d70b44d906a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x\" (UID: \"42444876-b7b4-44b7-b051-d70b44d906a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x"
Jul 28 21:38:47.293004 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:47.292983    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42444876-b7b4-44b7-b051-d70b44d906a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x\" (UID: \"42444876-b7b4-44b7-b051-d70b44d906a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x"
Jul 28 21:38:47.304379 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:47.304349    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rztl9\" (UniqueName: \"kubernetes.io/projected/42444876-b7b4-44b7-b051-d70b44d906a3-kube-api-access-rztl9\") pod \"payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x\" (UID: \"42444876-b7b4-44b7-b051-d70b44d906a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x"
Jul 28 21:38:47.336487 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:47.336455    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"]
Jul 28 21:38:47.337216 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:38:47.337187    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11dfd8a8_3adb_4e82_ac4d_1a23ee3f318c.slice/crio-ecf2d3ee364572975051e73c126e5957bafb0757dfbe4a3ec5b2f231fc8cebfe WatchSource:0}: Error finding container ecf2d3ee364572975051e73c126e5957bafb0757dfbe4a3ec5b2f231fc8cebfe: Status 404 returned error can't find the container with id ecf2d3ee364572975051e73c126e5957bafb0757dfbe4a3ec5b2f231fc8cebfe
Jul 28 21:38:47.377793 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:47.377760    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x"
Jul 28 21:38:47.518014 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:47.517989    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x"]
Jul 28 21:38:47.519584 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:38:47.519559    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42444876_b7b4_44b7_b051_d70b44d906a3.slice/crio-b27e4cac06c5fcbfdaebbbf050a23dd60de82847ecb8f6b2b62f0dc2ab7a4a1d WatchSource:0}: Error finding container b27e4cac06c5fcbfdaebbbf050a23dd60de82847ecb8f6b2b62f0dc2ab7a4a1d: Status 404 returned error can't find the container with id b27e4cac06c5fcbfdaebbbf050a23dd60de82847ecb8f6b2b62f0dc2ab7a4a1d
Jul 28 21:38:47.650288 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:47.650244    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x" event={"ID":"42444876-b7b4-44b7-b051-d70b44d906a3","Type":"ContainerStarted","Data":"b6e627c9b462a68c7b90db48dde03aeb68854a4c0cc6d2ea7a33f06b299f3f89"}
Jul 28 21:38:47.650495 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:47.650319    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x" event={"ID":"42444876-b7b4-44b7-b051-d70b44d906a3","Type":"ContainerStarted","Data":"b27e4cac06c5fcbfdaebbbf050a23dd60de82847ecb8f6b2b62f0dc2ab7a4a1d"}
Jul 28 21:38:47.650569 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:47.650533    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x"
Jul 28 21:38:47.652080 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:47.652048    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm" event={"ID":"11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c","Type":"ContainerStarted","Data":"ecf2d3ee364572975051e73c126e5957bafb0757dfbe4a3ec5b2f231fc8cebfe"}
Jul 28 21:38:47.667488 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:47.667431    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x" podStartSLOduration=0.667413205 podStartE2EDuration="667.413205ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:38:47.665104831 +0000 UTC m=+3082.889447822" watchObservedRunningTime="2026-07-28 21:38:47.667413205 +0000 UTC m=+3082.891756192"
Jul 28 21:38:48.660909 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:48.660863    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm" event={"ID":"11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c","Type":"ContainerStarted","Data":"54749e116a77c14baca460b83c2082d1c02ca879e9117b1e9a86b471114cf381"}
Jul 28 21:38:48.661431 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:48.661146    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"
Jul 28 21:38:54.672783 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:54.672749    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"
Jul 28 21:38:54.691848 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:54.691776    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm" podStartSLOduration=8.346809962 podStartE2EDuration="8.69176112s" totalImagesPullingTime="344.951158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:38:48.719780673 +0000 UTC m=+3083.944123660" watchObservedRunningTime="2026-07-28 21:38:54.69176112 +0000 UTC m=+3089.916104106"
Jul 28 21:38:55.365296 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:55.365265    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:38:58.663173 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:38:58.663143    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x"
Jul 28 21:39:17.929093 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:39:17.929055    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x_42444876-b7b4-44b7-b051-d70b44d906a3/payload-processing/0.log"
Jul 28 21:39:34.158579 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:39:34.158542    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:39:58.662239 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:39:58.662158    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:40:09.792947 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:09.792914    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl_d7720ded-b3d7-4810-852c-5d86c5f01893/payload-processing/0.log"
Jul 28 21:40:10.020020 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:10.019991    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x_42444876-b7b4-44b7-b051-d70b44d906a3/payload-processing/0.log"
Jul 28 21:40:13.264465 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:13.264432    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"]
Jul 28 21:40:13.268112 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:13.268094    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"
Jul 28 21:40:13.270401 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:13.270373    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-ea76b4-serving-cert\""
Jul 28 21:40:13.277291 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:13.277266    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"]
Jul 28 21:40:13.342457 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:13.342418    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlsql\" (UniqueName: \"kubernetes.io/projected/8b25814b-9a74-4328-bd96-d1522b7caa28-kube-api-access-qlsql\") pod \"maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz\" (UID: \"8b25814b-9a74-4328-bd96-d1522b7caa28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"
Jul 28 21:40:13.342457 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:13.342464    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b25814b-9a74-4328-bd96-d1522b7caa28-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz\" (UID: \"8b25814b-9a74-4328-bd96-d1522b7caa28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"
Jul 28 21:40:13.342714 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:13.342481    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b25814b-9a74-4328-bd96-d1522b7caa28-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz\" (UID: \"8b25814b-9a74-4328-bd96-d1522b7caa28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"
Jul 28 21:40:13.443589 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:13.443553    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b25814b-9a74-4328-bd96-d1522b7caa28-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz\" (UID: \"8b25814b-9a74-4328-bd96-d1522b7caa28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"
Jul 28 21:40:13.443589 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:13.443589    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b25814b-9a74-4328-bd96-d1522b7caa28-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz\" (UID: \"8b25814b-9a74-4328-bd96-d1522b7caa28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"
Jul 28 21:40:13.443829 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:13.443685    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlsql\" (UniqueName: \"kubernetes.io/projected/8b25814b-9a74-4328-bd96-d1522b7caa28-kube-api-access-qlsql\") pod \"maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz\" (UID: \"8b25814b-9a74-4328-bd96-d1522b7caa28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"
Jul 28 21:40:13.444341 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:13.444321    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b25814b-9a74-4328-bd96-d1522b7caa28-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz\" (UID: \"8b25814b-9a74-4328-bd96-d1522b7caa28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"
Jul 28 21:40:13.445999 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:13.445980    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b25814b-9a74-4328-bd96-d1522b7caa28-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz\" (UID: \"8b25814b-9a74-4328-bd96-d1522b7caa28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"
Jul 28 21:40:13.450436 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:13.450413    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlsql\" (UniqueName: \"kubernetes.io/projected/8b25814b-9a74-4328-bd96-d1522b7caa28-kube-api-access-qlsql\") pod \"maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz\" (UID: \"8b25814b-9a74-4328-bd96-d1522b7caa28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"
Jul 28 21:40:13.580117 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:13.580024    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"
Jul 28 21:40:13.718390 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:13.718342    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"]
Jul 28 21:40:13.720467 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:40:13.720432    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b25814b_9a74_4328_bd96_d1522b7caa28.slice/crio-b7d3592903cc49ccc5e35bbb647fc0514f74d4bb4b1c0171f36ce1dad5e47858 WatchSource:0}: Error finding container b7d3592903cc49ccc5e35bbb647fc0514f74d4bb4b1c0171f36ce1dad5e47858: Status 404 returned error can't find the container with id b7d3592903cc49ccc5e35bbb647fc0514f74d4bb4b1c0171f36ce1dad5e47858
Jul 28 21:40:14.173321 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:14.173275    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz" event={"ID":"8b25814b-9a74-4328-bd96-d1522b7caa28","Type":"ContainerStarted","Data":"b7d3592903cc49ccc5e35bbb647fc0514f74d4bb4b1c0171f36ce1dad5e47858"}
Jul 28 21:40:15.180126 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:15.180082    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz" event={"ID":"8b25814b-9a74-4328-bd96-d1522b7caa28","Type":"ContainerStarted","Data":"7308577aff084a139f3d88a845375dbfaa03eeced7cf2b980ed99cf04b051654"}
Jul 28 21:40:15.180599 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:15.180162    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"
Jul 28 21:40:15.198322 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:15.198268    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz" podStartSLOduration=1.692304432 podStartE2EDuration="2.19825061s" totalImagesPullingTime="505.946178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:40:15.197934008 +0000 UTC m=+3170.422277015" watchObservedRunningTime="2026-07-28 21:40:15.19825061 +0000 UTC m=+3170.422593611"
Jul 28 21:40:18.066805 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:18.066765    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:40:21.192287 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:21.192249    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"
Jul 28 21:40:38.832836 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:38.832795    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ea76b4-vv2bx"]
Jul 28 21:40:38.836375 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:38.836359    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ea76b4-vv2bx"
Jul 28 21:40:38.838088 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:38.838067    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w22xz\""
Jul 28 21:40:38.842910 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:38.842889    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ea76b4-vv2bx"]
Jul 28 21:40:38.974412 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:38.974372    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe366b2-3dbd-4352-80de-7207069b2470-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ea76b4-vv2bx\" (UID: \"1fe366b2-3dbd-4352-80de-7207069b2470\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ea76b4-vv2bx"
Jul 28 21:40:39.075179 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:39.075134    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe366b2-3dbd-4352-80de-7207069b2470-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ea76b4-vv2bx\" (UID: \"1fe366b2-3dbd-4352-80de-7207069b2470\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ea76b4-vv2bx"
Jul 28 21:40:39.075858 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:39.075828    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe366b2-3dbd-4352-80de-7207069b2470-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ea76b4-vv2bx\" (UID: \"1fe366b2-3dbd-4352-80de-7207069b2470\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ea76b4-vv2bx"
Jul 28 21:40:39.149424 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:39.149338    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ea76b4-vv2bx"
Jul 28 21:40:39.288703 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:39.288670    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ea76b4-vv2bx"]
Jul 28 21:40:39.289957 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:40:39.289926    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe366b2_3dbd_4352_80de_7207069b2470.slice/crio-f0b7b29985fd4d76c8acf2a493d81620b235c03409aaa920f1077b3215eddced WatchSource:0}: Error finding container f0b7b29985fd4d76c8acf2a493d81620b235c03409aaa920f1077b3215eddced: Status 404 returned error can't find the container with id f0b7b29985fd4d76c8acf2a493d81620b235c03409aaa920f1077b3215eddced
Jul 28 21:40:39.317414 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:39.317386    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ea76b4-vv2bx" event={"ID":"1fe366b2-3dbd-4352-80de-7207069b2470","Type":"ContainerStarted","Data":"f0b7b29985fd4d76c8acf2a493d81620b235c03409aaa920f1077b3215eddced"}
Jul 28 21:40:40.324227 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:40.324133    2565 generic.go:356] "Generic (PLEG): container finished" podID="1fe366b2-3dbd-4352-80de-7207069b2470" containerID="6c070804ffcb8a36084ec6319b2ae1ab1f2923c16b7c8ee232b5a59844e27eb8" exitCode=0
Jul 28 21:40:40.324616 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:40.324220    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ea76b4-vv2bx" event={"ID":"1fe366b2-3dbd-4352-80de-7207069b2470","Type":"ContainerDied","Data":"6c070804ffcb8a36084ec6319b2ae1ab1f2923c16b7c8ee232b5a59844e27eb8"}
Jul 28 21:40:41.466072 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:41.466046    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ea76b4-vv2bx"
Jul 28 21:40:41.596169 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:41.596082    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe366b2-3dbd-4352-80de-7207069b2470-maas-api-service-ca\") pod \"1fe366b2-3dbd-4352-80de-7207069b2470\" (UID: \"1fe366b2-3dbd-4352-80de-7207069b2470\") "
Jul 28 21:40:41.596469 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:41.596446    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe366b2-3dbd-4352-80de-7207069b2470-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1fe366b2-3dbd-4352-80de-7207069b2470" (UID: "1fe366b2-3dbd-4352-80de-7207069b2470"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:40:41.697758 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:41.697723    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe366b2-3dbd-4352-80de-7207069b2470-maas-api-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:40:42.337057 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:42.337027    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ea76b4-vv2bx"
Jul 28 21:40:42.337254 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:42.337061    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ea76b4-vv2bx" event={"ID":"1fe366b2-3dbd-4352-80de-7207069b2470","Type":"ContainerDied","Data":"f0b7b29985fd4d76c8acf2a493d81620b235c03409aaa920f1077b3215eddced"}
Jul 28 21:40:42.337254 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:42.337109    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0b7b29985fd4d76c8acf2a493d81620b235c03409aaa920f1077b3215eddced"
Jul 28 21:40:43.871033 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:43.870999    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"]
Jul 28 21:40:43.871534 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:43.871224    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz" podUID="8b25814b-9a74-4328-bd96-d1522b7caa28" containerName="maas-api" containerID="cri-o://7308577aff084a139f3d88a845375dbfaa03eeced7cf2b980ed99cf04b051654" gracePeriod=30
Jul 28 21:40:44.064690 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.064654    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ea76b4-vv2bx"]
Jul 28 21:40:44.068649 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.068599    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ea76b4-vv2bx"]
Jul 28 21:40:44.128376 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.128313    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"
Jul 28 21:40:44.220467 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.220423    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b25814b-9a74-4328-bd96-d1522b7caa28-openshift-service-ca\") pod \"8b25814b-9a74-4328-bd96-d1522b7caa28\" (UID: \"8b25814b-9a74-4328-bd96-d1522b7caa28\") "
Jul 28 21:40:44.220467 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.220470    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlsql\" (UniqueName: \"kubernetes.io/projected/8b25814b-9a74-4328-bd96-d1522b7caa28-kube-api-access-qlsql\") pod \"8b25814b-9a74-4328-bd96-d1522b7caa28\" (UID: \"8b25814b-9a74-4328-bd96-d1522b7caa28\") "
Jul 28 21:40:44.220778 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.220503    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b25814b-9a74-4328-bd96-d1522b7caa28-maas-api-tls\") pod \"8b25814b-9a74-4328-bd96-d1522b7caa28\" (UID: \"8b25814b-9a74-4328-bd96-d1522b7caa28\") "
Jul 28 21:40:44.220921 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.220896    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b25814b-9a74-4328-bd96-d1522b7caa28-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8b25814b-9a74-4328-bd96-d1522b7caa28" (UID: "8b25814b-9a74-4328-bd96-d1522b7caa28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:40:44.222645 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.222609    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b25814b-9a74-4328-bd96-d1522b7caa28-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8b25814b-9a74-4328-bd96-d1522b7caa28" (UID: "8b25814b-9a74-4328-bd96-d1522b7caa28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:40:44.222806 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.222787    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b25814b-9a74-4328-bd96-d1522b7caa28-kube-api-access-qlsql" (OuterVolumeSpecName: "kube-api-access-qlsql") pod "8b25814b-9a74-4328-bd96-d1522b7caa28" (UID: "8b25814b-9a74-4328-bd96-d1522b7caa28"). InnerVolumeSpecName "kube-api-access-qlsql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:40:44.321522 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.321480    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b25814b-9a74-4328-bd96-d1522b7caa28-openshift-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:40:44.321522 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.321511    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlsql\" (UniqueName: \"kubernetes.io/projected/8b25814b-9a74-4328-bd96-d1522b7caa28-kube-api-access-qlsql\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:40:44.321522 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.321522    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b25814b-9a74-4328-bd96-d1522b7caa28-maas-api-tls\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:40:44.349369 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.349336    2565 generic.go:356] "Generic (PLEG): container finished" podID="8b25814b-9a74-4328-bd96-d1522b7caa28" containerID="7308577aff084a139f3d88a845375dbfaa03eeced7cf2b980ed99cf04b051654" exitCode=0
Jul 28 21:40:44.349543 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.349387    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"
Jul 28 21:40:44.349543 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.349423    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz" event={"ID":"8b25814b-9a74-4328-bd96-d1522b7caa28","Type":"ContainerDied","Data":"7308577aff084a139f3d88a845375dbfaa03eeced7cf2b980ed99cf04b051654"}
Jul 28 21:40:44.349543 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.349478    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz" event={"ID":"8b25814b-9a74-4328-bd96-d1522b7caa28","Type":"ContainerDied","Data":"b7d3592903cc49ccc5e35bbb647fc0514f74d4bb4b1c0171f36ce1dad5e47858"}
Jul 28 21:40:44.349543 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.349506    2565 scope.go:117] "RemoveContainer" containerID="7308577aff084a139f3d88a845375dbfaa03eeced7cf2b980ed99cf04b051654"
Jul 28 21:40:44.359810 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.359791    2565 scope.go:117] "RemoveContainer" containerID="7308577aff084a139f3d88a845375dbfaa03eeced7cf2b980ed99cf04b051654"
Jul 28 21:40:44.360074 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:40:44.360058    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7308577aff084a139f3d88a845375dbfaa03eeced7cf2b980ed99cf04b051654\": container with ID starting with 7308577aff084a139f3d88a845375dbfaa03eeced7cf2b980ed99cf04b051654 not found: ID does not exist" containerID="7308577aff084a139f3d88a845375dbfaa03eeced7cf2b980ed99cf04b051654"
Jul 28 21:40:44.360125 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.360083    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7308577aff084a139f3d88a845375dbfaa03eeced7cf2b980ed99cf04b051654"} err="failed to get container status \"7308577aff084a139f3d88a845375dbfaa03eeced7cf2b980ed99cf04b051654\": rpc error: code = NotFound desc = could not find container \"7308577aff084a139f3d88a845375dbfaa03eeced7cf2b980ed99cf04b051654\": container with ID starting with 7308577aff084a139f3d88a845375dbfaa03eeced7cf2b980ed99cf04b051654 not found: ID does not exist"
Jul 28 21:40:44.378628 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.378575    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"]
Jul 28 21:40:44.383108 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:44.383084    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ea76b4-f5b6d64bf-ksdrz"]
Jul 28 21:40:45.332327 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:45.332296    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fe366b2-3dbd-4352-80de-7207069b2470" path="/var/lib/kubelet/pods/1fe366b2-3dbd-4352-80de-7207069b2470/volumes"
Jul 28 21:40:45.332731 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:45.332681    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b25814b-9a74-4328-bd96-d1522b7caa28" path="/var/lib/kubelet/pods/8b25814b-9a74-4328-bd96-d1522b7caa28/volumes"
Jul 28 21:40:47.863406 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:47.863369    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:40:54.001245 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:54.001205    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf1484-dtgn6"]
Jul 28 21:40:54.001754 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:54.001605    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b25814b-9a74-4328-bd96-d1522b7caa28" containerName="maas-api"
Jul 28 21:40:54.001754 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:54.001616    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b25814b-9a74-4328-bd96-d1522b7caa28" containerName="maas-api"
Jul 28 21:40:54.001754 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:54.001631    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe366b2-3dbd-4352-80de-7207069b2470" containerName="revoke-keys"
Jul 28 21:40:54.001754 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:54.001650    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe366b2-3dbd-4352-80de-7207069b2470" containerName="revoke-keys"
Jul 28 21:40:54.001754 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:54.001714    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b25814b-9a74-4328-bd96-d1522b7caa28" containerName="maas-api"
Jul 28 21:40:54.001754 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:54.001725    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fe366b2-3dbd-4352-80de-7207069b2470" containerName="revoke-keys"
Jul 28 21:40:54.006397 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:54.006379    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf1484-dtgn6"
Jul 28 21:40:54.008816 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:54.008464    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w22xz\""
Jul 28 21:40:54.012254 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:54.012228    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf1484-dtgn6"]
Jul 28 21:40:54.107923 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:54.107869    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cabb6be5-5099-4f02-afd1-dbf3b910e430-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf1484-dtgn6\" (UID: \"cabb6be5-5099-4f02-afd1-dbf3b910e430\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf1484-dtgn6"
Jul 28 21:40:54.209212 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:54.209179    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cabb6be5-5099-4f02-afd1-dbf3b910e430-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf1484-dtgn6\" (UID: \"cabb6be5-5099-4f02-afd1-dbf3b910e430\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf1484-dtgn6"
Jul 28 21:40:54.209862 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:54.209840    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cabb6be5-5099-4f02-afd1-dbf3b910e430-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cf1484-dtgn6\" (UID: \"cabb6be5-5099-4f02-afd1-dbf3b910e430\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf1484-dtgn6"
Jul 28 21:40:54.319278 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:54.319190    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf1484-dtgn6"
Jul 28 21:40:54.453593 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:54.453568    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf1484-dtgn6"]
Jul 28 21:40:54.454794 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:40:54.454771    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabb6be5_5099_4f02_afd1_dbf3b910e430.slice/crio-04bf1deaeba6747a5d167993c206e1b5fd94ae2f9a9b0fcc6a75cae984d8fa62 WatchSource:0}: Error finding container 04bf1deaeba6747a5d167993c206e1b5fd94ae2f9a9b0fcc6a75cae984d8fa62: Status 404 returned error can't find the container with id 04bf1deaeba6747a5d167993c206e1b5fd94ae2f9a9b0fcc6a75cae984d8fa62
Jul 28 21:40:55.419605 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:55.419511    2565 generic.go:356] "Generic (PLEG): container finished" podID="cabb6be5-5099-4f02-afd1-dbf3b910e430" containerID="822fba4b6124277ccaa6fd8a47d9b6ecaab06542b6d887cd91975ee545a13b02" exitCode=0
Jul 28 21:40:55.419605 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:55.419557    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf1484-dtgn6" event={"ID":"cabb6be5-5099-4f02-afd1-dbf3b910e430","Type":"ContainerDied","Data":"822fba4b6124277ccaa6fd8a47d9b6ecaab06542b6d887cd91975ee545a13b02"}
Jul 28 21:40:55.420052 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:55.419612    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf1484-dtgn6" event={"ID":"cabb6be5-5099-4f02-afd1-dbf3b910e430","Type":"ContainerStarted","Data":"04bf1deaeba6747a5d167993c206e1b5fd94ae2f9a9b0fcc6a75cae984d8fa62"}
Jul 28 21:40:56.565730 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:56.565704    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf1484-dtgn6"
Jul 28 21:40:56.731218 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:56.731189    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cabb6be5-5099-4f02-afd1-dbf3b910e430-maas-api-service-ca\") pod \"cabb6be5-5099-4f02-afd1-dbf3b910e430\" (UID: \"cabb6be5-5099-4f02-afd1-dbf3b910e430\") "
Jul 28 21:40:56.731546 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:56.731524    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabb6be5-5099-4f02-afd1-dbf3b910e430-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cabb6be5-5099-4f02-afd1-dbf3b910e430" (UID: "cabb6be5-5099-4f02-afd1-dbf3b910e430"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:40:56.832424 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:56.832382    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cabb6be5-5099-4f02-afd1-dbf3b910e430-maas-api-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:40:57.432072 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:57.432032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf1484-dtgn6" event={"ID":"cabb6be5-5099-4f02-afd1-dbf3b910e430","Type":"ContainerDied","Data":"04bf1deaeba6747a5d167993c206e1b5fd94ae2f9a9b0fcc6a75cae984d8fa62"}
Jul 28 21:40:57.432072 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:57.432066    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf1484-dtgn6"
Jul 28 21:40:57.432290 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:40:57.432094    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04bf1deaeba6747a5d167993c206e1b5fd94ae2f9a9b0fcc6a75cae984d8fa62"
Jul 28 21:41:08.766273 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:08.766237    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:41:09.786027 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:09.785986    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"]
Jul 28 21:41:09.786439 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:09.786249    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz" podUID="7115d06e-9be9-4b77-a390-87beaa33cb92" containerName="maas-api" containerID="cri-o://0a48e71d226d1196fed553154f04815363a6d68825735bd094d48eaf072ce14b" gracePeriod=30
Jul 28 21:41:09.850404 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:09.850367    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl"]
Jul 28 21:41:09.850690 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:09.850661    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl" podUID="d7720ded-b3d7-4810-852c-5d86c5f01893" containerName="payload-processing" containerID="cri-o://48085b7dcde9f6e9782161d269794f4694ede0a89ed51e3d3aff85eb6f67dca5" gracePeriod=30
Jul 28 21:41:09.981178 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:09.981108    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf1484-dtgn6"]
Jul 28 21:41:09.983580 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:09.983550    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cf1484-dtgn6"]
Jul 28 21:41:10.104031 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.104003    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"
Jul 28 21:41:10.131327 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.131303    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl"
Jul 28 21:41:10.134498 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.134470    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7720ded-b3d7-4810-852c-5d86c5f01893-plugins-config-volume\") pod \"d7720ded-b3d7-4810-852c-5d86c5f01893\" (UID: \"d7720ded-b3d7-4810-852c-5d86c5f01893\") "
Jul 28 21:41:10.134667 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.134513    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7115d06e-9be9-4b77-a390-87beaa33cb92-openshift-service-ca\") pod \"7115d06e-9be9-4b77-a390-87beaa33cb92\" (UID: \"7115d06e-9be9-4b77-a390-87beaa33cb92\") "
Jul 28 21:41:10.134667 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.134561    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcdqv\" (UniqueName: \"kubernetes.io/projected/d7720ded-b3d7-4810-852c-5d86c5f01893-kube-api-access-jcdqv\") pod \"d7720ded-b3d7-4810-852c-5d86c5f01893\" (UID: \"d7720ded-b3d7-4810-852c-5d86c5f01893\") "
Jul 28 21:41:10.134818 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.134666    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7115d06e-9be9-4b77-a390-87beaa33cb92-maas-api-tls\") pod \"7115d06e-9be9-4b77-a390-87beaa33cb92\" (UID: \"7115d06e-9be9-4b77-a390-87beaa33cb92\") "
Jul 28 21:41:10.134818 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.134706    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dbnf\" (UniqueName: \"kubernetes.io/projected/7115d06e-9be9-4b77-a390-87beaa33cb92-kube-api-access-8dbnf\") pod \"7115d06e-9be9-4b77-a390-87beaa33cb92\" (UID: \"7115d06e-9be9-4b77-a390-87beaa33cb92\") "
Jul 28 21:41:10.134927 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.134905    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7720ded-b3d7-4810-852c-5d86c5f01893-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d7720ded-b3d7-4810-852c-5d86c5f01893" (UID: "d7720ded-b3d7-4810-852c-5d86c5f01893"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:41:10.134987 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.134971    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7720ded-b3d7-4810-852c-5d86c5f01893-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:41:10.135045 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.134993    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7115d06e-9be9-4b77-a390-87beaa33cb92-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7115d06e-9be9-4b77-a390-87beaa33cb92" (UID: "7115d06e-9be9-4b77-a390-87beaa33cb92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:41:10.136877 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.136851    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7115d06e-9be9-4b77-a390-87beaa33cb92-kube-api-access-8dbnf" (OuterVolumeSpecName: "kube-api-access-8dbnf") pod "7115d06e-9be9-4b77-a390-87beaa33cb92" (UID: "7115d06e-9be9-4b77-a390-87beaa33cb92"). InnerVolumeSpecName "kube-api-access-8dbnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:41:10.136980 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.136908    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7115d06e-9be9-4b77-a390-87beaa33cb92-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7115d06e-9be9-4b77-a390-87beaa33cb92" (UID: "7115d06e-9be9-4b77-a390-87beaa33cb92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:41:10.137056 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.137002    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7720ded-b3d7-4810-852c-5d86c5f01893-kube-api-access-jcdqv" (OuterVolumeSpecName: "kube-api-access-jcdqv") pod "d7720ded-b3d7-4810-852c-5d86c5f01893" (UID: "d7720ded-b3d7-4810-852c-5d86c5f01893"). InnerVolumeSpecName "kube-api-access-jcdqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:41:10.235495 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.235459    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7115d06e-9be9-4b77-a390-87beaa33cb92-maas-api-tls\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:41:10.235495 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.235494    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dbnf\" (UniqueName: \"kubernetes.io/projected/7115d06e-9be9-4b77-a390-87beaa33cb92-kube-api-access-8dbnf\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:41:10.235768 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.235507    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7115d06e-9be9-4b77-a390-87beaa33cb92-openshift-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:41:10.235768 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.235520    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcdqv\" (UniqueName: \"kubernetes.io/projected/d7720ded-b3d7-4810-852c-5d86c5f01893-kube-api-access-jcdqv\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:41:10.508033 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.508000    2565 generic.go:356] "Generic (PLEG): container finished" podID="7115d06e-9be9-4b77-a390-87beaa33cb92" containerID="0a48e71d226d1196fed553154f04815363a6d68825735bd094d48eaf072ce14b" exitCode=0
Jul 28 21:41:10.508240 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.508044    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"
Jul 28 21:41:10.508240 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.508092    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz" event={"ID":"7115d06e-9be9-4b77-a390-87beaa33cb92","Type":"ContainerDied","Data":"0a48e71d226d1196fed553154f04815363a6d68825735bd094d48eaf072ce14b"}
Jul 28 21:41:10.508240 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.508152    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz" event={"ID":"7115d06e-9be9-4b77-a390-87beaa33cb92","Type":"ContainerDied","Data":"1d496c6fb31bdfefe7d165723f01c72dfac78d30fc910bdbdf9d5308f68c7b0c"}
Jul 28 21:41:10.508240 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.508181    2565 scope.go:117] "RemoveContainer" containerID="0a48e71d226d1196fed553154f04815363a6d68825735bd094d48eaf072ce14b"
Jul 28 21:41:10.509577 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.509541    2565 generic.go:356] "Generic (PLEG): container finished" podID="d7720ded-b3d7-4810-852c-5d86c5f01893" containerID="48085b7dcde9f6e9782161d269794f4694ede0a89ed51e3d3aff85eb6f67dca5" exitCode=0
Jul 28 21:41:10.509577 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.509566    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl" event={"ID":"d7720ded-b3d7-4810-852c-5d86c5f01893","Type":"ContainerDied","Data":"48085b7dcde9f6e9782161d269794f4694ede0a89ed51e3d3aff85eb6f67dca5"}
Jul 28 21:41:10.509749 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.509597    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl"
Jul 28 21:41:10.509749 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.509617    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl" event={"ID":"d7720ded-b3d7-4810-852c-5d86c5f01893","Type":"ContainerDied","Data":"04327cb5b2588b84c8ad6614c937c26705ec6163f2d0abfde3e20874fd589a0f"}
Jul 28 21:41:10.521461 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.521370    2565 scope.go:117] "RemoveContainer" containerID="0a48e71d226d1196fed553154f04815363a6d68825735bd094d48eaf072ce14b"
Jul 28 21:41:10.521746 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:41:10.521721    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a48e71d226d1196fed553154f04815363a6d68825735bd094d48eaf072ce14b\": container with ID starting with 0a48e71d226d1196fed553154f04815363a6d68825735bd094d48eaf072ce14b not found: ID does not exist" containerID="0a48e71d226d1196fed553154f04815363a6d68825735bd094d48eaf072ce14b"
Jul 28 21:41:10.521843 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.521784    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a48e71d226d1196fed553154f04815363a6d68825735bd094d48eaf072ce14b"} err="failed to get container status \"0a48e71d226d1196fed553154f04815363a6d68825735bd094d48eaf072ce14b\": rpc error: code = NotFound desc = could not find container \"0a48e71d226d1196fed553154f04815363a6d68825735bd094d48eaf072ce14b\": container with ID starting with 0a48e71d226d1196fed553154f04815363a6d68825735bd094d48eaf072ce14b not found: ID does not exist"
Jul 28 21:41:10.521843 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.521811    2565 scope.go:117] "RemoveContainer" containerID="48085b7dcde9f6e9782161d269794f4694ede0a89ed51e3d3aff85eb6f67dca5"
Jul 28 21:41:10.530510 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.530494    2565 scope.go:117] "RemoveContainer" containerID="48085b7dcde9f6e9782161d269794f4694ede0a89ed51e3d3aff85eb6f67dca5"
Jul 28 21:41:10.530793 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:41:10.530772    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48085b7dcde9f6e9782161d269794f4694ede0a89ed51e3d3aff85eb6f67dca5\": container with ID starting with 48085b7dcde9f6e9782161d269794f4694ede0a89ed51e3d3aff85eb6f67dca5 not found: ID does not exist" containerID="48085b7dcde9f6e9782161d269794f4694ede0a89ed51e3d3aff85eb6f67dca5"
Jul 28 21:41:10.530866 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.530800    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48085b7dcde9f6e9782161d269794f4694ede0a89ed51e3d3aff85eb6f67dca5"} err="failed to get container status \"48085b7dcde9f6e9782161d269794f4694ede0a89ed51e3d3aff85eb6f67dca5\": rpc error: code = NotFound desc = could not find container \"48085b7dcde9f6e9782161d269794f4694ede0a89ed51e3d3aff85eb6f67dca5\": container with ID starting with 48085b7dcde9f6e9782161d269794f4694ede0a89ed51e3d3aff85eb6f67dca5 not found: ID does not exist"
Jul 28 21:41:10.532384 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.532362    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"]
Jul 28 21:41:10.535697 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.535675    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-cf1484-bb4cf4d89-whpsz"]
Jul 28 21:41:10.544365 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.544346    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl"]
Jul 28 21:41:10.547853 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:10.547834    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cf1484-6f76599d5d-gcmzl"]
Jul 28 21:41:11.331422 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:11.331389    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7115d06e-9be9-4b77-a390-87beaa33cb92" path="/var/lib/kubelet/pods/7115d06e-9be9-4b77-a390-87beaa33cb92/volumes"
Jul 28 21:41:11.331830 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:11.331750    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cabb6be5-5099-4f02-afd1-dbf3b910e430" path="/var/lib/kubelet/pods/cabb6be5-5099-4f02-afd1-dbf3b910e430/volumes"
Jul 28 21:41:11.332097 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:11.332083    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7720ded-b3d7-4810-852c-5d86c5f01893" path="/var/lib/kubelet/pods/d7720ded-b3d7-4810-852c-5d86c5f01893/volumes"
Jul 28 21:41:19.168502 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:19.168365    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:41:59.286072 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:59.286028    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d69596-6vtcf"]
Jul 28 21:41:59.286554 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:59.286417    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7115d06e-9be9-4b77-a390-87beaa33cb92" containerName="maas-api"
Jul 28 21:41:59.286554 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:59.286428    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7115d06e-9be9-4b77-a390-87beaa33cb92" containerName="maas-api"
Jul 28 21:41:59.286554 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:59.286449    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cabb6be5-5099-4f02-afd1-dbf3b910e430" containerName="revoke-keys"
Jul 28 21:41:59.286554 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:59.286455    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cabb6be5-5099-4f02-afd1-dbf3b910e430" containerName="revoke-keys"
Jul 28 21:41:59.286554 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:59.286462    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7720ded-b3d7-4810-852c-5d86c5f01893" containerName="payload-processing"
Jul 28 21:41:59.286554 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:59.286469    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7720ded-b3d7-4810-852c-5d86c5f01893" containerName="payload-processing"
Jul 28 21:41:59.286554 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:59.286536    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cabb6be5-5099-4f02-afd1-dbf3b910e430" containerName="revoke-keys"
Jul 28 21:41:59.286554 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:59.286544    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7115d06e-9be9-4b77-a390-87beaa33cb92" containerName="maas-api"
Jul 28 21:41:59.286554 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:59.286554    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7720ded-b3d7-4810-852c-5d86c5f01893" containerName="payload-processing"
Jul 28 21:41:59.289666 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:59.289647    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d69596-6vtcf"
Jul 28 21:41:59.291906 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:59.291884    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w22xz\""
Jul 28 21:41:59.297768 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:59.297739    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d69596-6vtcf"]
Jul 28 21:41:59.456228 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:59.456185    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f1499a-16f1-4cb1-95ed-bcec74d0ca29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d69596-6vtcf\" (UID: \"a2f1499a-16f1-4cb1-95ed-bcec74d0ca29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d69596-6vtcf"
Jul 28 21:41:59.557689 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:59.557587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f1499a-16f1-4cb1-95ed-bcec74d0ca29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d69596-6vtcf\" (UID: \"a2f1499a-16f1-4cb1-95ed-bcec74d0ca29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d69596-6vtcf"
Jul 28 21:41:59.558220 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:59.558199    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f1499a-16f1-4cb1-95ed-bcec74d0ca29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d69596-6vtcf\" (UID: \"a2f1499a-16f1-4cb1-95ed-bcec74d0ca29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d69596-6vtcf"
Jul 28 21:41:59.600780 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:59.600739    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d69596-6vtcf"
Jul 28 21:41:59.734926 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:59.734897    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d69596-6vtcf"]
Jul 28 21:41:59.737204 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:41:59.737177    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f1499a_16f1_4cb1_95ed_bcec74d0ca29.slice/crio-2ba36bae74dfd6d7f862605398234060163bf78b49ea955813bb7de718b29fb5 WatchSource:0}: Error finding container 2ba36bae74dfd6d7f862605398234060163bf78b49ea955813bb7de718b29fb5: Status 404 returned error can't find the container with id 2ba36bae74dfd6d7f862605398234060163bf78b49ea955813bb7de718b29fb5
Jul 28 21:41:59.794153 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:41:59.794118    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d69596-6vtcf" event={"ID":"a2f1499a-16f1-4cb1-95ed-bcec74d0ca29","Type":"ContainerStarted","Data":"2ba36bae74dfd6d7f862605398234060163bf78b49ea955813bb7de718b29fb5"}
Jul 28 21:42:00.801932 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:00.801849    2565 generic.go:356] "Generic (PLEG): container finished" podID="a2f1499a-16f1-4cb1-95ed-bcec74d0ca29" containerID="0a88a2254815e4ba2eac81231669555aef32b693ce2fd21f8854f5618a7a177e" exitCode=0
Jul 28 21:42:00.801932 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:00.801899    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d69596-6vtcf" event={"ID":"a2f1499a-16f1-4cb1-95ed-bcec74d0ca29","Type":"ContainerDied","Data":"0a88a2254815e4ba2eac81231669555aef32b693ce2fd21f8854f5618a7a177e"}
Jul 28 21:42:01.943957 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:01.943934    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d69596-6vtcf"
Jul 28 21:42:02.079280 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:02.079193    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f1499a-16f1-4cb1-95ed-bcec74d0ca29-maas-api-service-ca\") pod \"a2f1499a-16f1-4cb1-95ed-bcec74d0ca29\" (UID: \"a2f1499a-16f1-4cb1-95ed-bcec74d0ca29\") "
Jul 28 21:42:02.079574 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:02.079549    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f1499a-16f1-4cb1-95ed-bcec74d0ca29-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a2f1499a-16f1-4cb1-95ed-bcec74d0ca29" (UID: "a2f1499a-16f1-4cb1-95ed-bcec74d0ca29"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:42:02.180536 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:02.180502    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2f1499a-16f1-4cb1-95ed-bcec74d0ca29-maas-api-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:42:02.814614 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:02.814589    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d69596-6vtcf"
Jul 28 21:42:02.814805 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:02.814586    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d69596-6vtcf" event={"ID":"a2f1499a-16f1-4cb1-95ed-bcec74d0ca29","Type":"ContainerDied","Data":"2ba36bae74dfd6d7f862605398234060163bf78b49ea955813bb7de718b29fb5"}
Jul 28 21:42:02.814805 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:02.814720    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ba36bae74dfd6d7f862605398234060163bf78b49ea955813bb7de718b29fb5"
Jul 28 21:42:09.796615 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:09.796576    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"]
Jul 28 21:42:09.797098 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:09.796866    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm" podUID="11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c" containerName="maas-api" containerID="cri-o://54749e116a77c14baca460b83c2082d1c02ca879e9117b1e9a86b471114cf381" gracePeriod=30
Jul 28 21:42:09.852698 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:09.852664    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x"]
Jul 28 21:42:09.852906 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:09.852876    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x" podUID="42444876-b7b4-44b7-b051-d70b44d906a3" containerName="payload-processing" containerID="cri-o://b6e627c9b462a68c7b90db48dde03aeb68854a4c0cc6d2ea7a33f06b299f3f89" gracePeriod=30
Jul 28 21:42:09.995914 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:09.995866    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d69596-6vtcf"]
Jul 28 21:42:09.998769 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:09.998735    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d69596-6vtcf"]
Jul 28 21:42:10.103034 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.103006    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"
Jul 28 21:42:10.137327 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.137302    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x"
Jul 28 21:42:10.156491 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.156462    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-maas-api-tls\") pod \"11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c\" (UID: \"11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c\") "
Jul 28 21:42:10.156699 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.156537    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rztl9\" (UniqueName: \"kubernetes.io/projected/42444876-b7b4-44b7-b051-d70b44d906a3-kube-api-access-rztl9\") pod \"42444876-b7b4-44b7-b051-d70b44d906a3\" (UID: \"42444876-b7b4-44b7-b051-d70b44d906a3\") "
Jul 28 21:42:10.156699 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.156591    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42444876-b7b4-44b7-b051-d70b44d906a3-plugins-config-volume\") pod \"42444876-b7b4-44b7-b051-d70b44d906a3\" (UID: \"42444876-b7b4-44b7-b051-d70b44d906a3\") "
Jul 28 21:42:10.156699 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.156618    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-openshift-service-ca\") pod \"11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c\" (UID: \"11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c\") "
Jul 28 21:42:10.156699 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.156682    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flt6k\" (UniqueName: \"kubernetes.io/projected/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-kube-api-access-flt6k\") pod \"11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c\" (UID: \"11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c\") "
Jul 28 21:42:10.157369 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.157003    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42444876-b7b4-44b7-b051-d70b44d906a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42444876-b7b4-44b7-b051-d70b44d906a3" (UID: "42444876-b7b4-44b7-b051-d70b44d906a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:42:10.157369 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.157071    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c" (UID: "11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:42:10.158813 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.158787    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42444876-b7b4-44b7-b051-d70b44d906a3-kube-api-access-rztl9" (OuterVolumeSpecName: "kube-api-access-rztl9") pod "42444876-b7b4-44b7-b051-d70b44d906a3" (UID: "42444876-b7b4-44b7-b051-d70b44d906a3"). InnerVolumeSpecName "kube-api-access-rztl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:42:10.159256 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.159234    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c" (UID: "11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:42:10.159256 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.159246    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-kube-api-access-flt6k" (OuterVolumeSpecName: "kube-api-access-flt6k") pod "11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c" (UID: "11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c"). InnerVolumeSpecName "kube-api-access-flt6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:42:10.258169 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.258131    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-maas-api-tls\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:42:10.258169 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.258164    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rztl9\" (UniqueName: \"kubernetes.io/projected/42444876-b7b4-44b7-b051-d70b44d906a3-kube-api-access-rztl9\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:42:10.258169 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.258174    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42444876-b7b4-44b7-b051-d70b44d906a3-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:42:10.258414 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.258186    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-openshift-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:42:10.258414 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.258197    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flt6k\" (UniqueName: \"kubernetes.io/projected/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c-kube-api-access-flt6k\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:42:10.867011 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.866971    2565 generic.go:356] "Generic (PLEG): container finished" podID="11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c" containerID="54749e116a77c14baca460b83c2082d1c02ca879e9117b1e9a86b471114cf381" exitCode=0
Jul 28 21:42:10.867506 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.867026    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"
Jul 28 21:42:10.867506 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.867055    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm" event={"ID":"11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c","Type":"ContainerDied","Data":"54749e116a77c14baca460b83c2082d1c02ca879e9117b1e9a86b471114cf381"}
Jul 28 21:42:10.867506 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.867126    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm" event={"ID":"11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c","Type":"ContainerDied","Data":"ecf2d3ee364572975051e73c126e5957bafb0757dfbe4a3ec5b2f231fc8cebfe"}
Jul 28 21:42:10.867506 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.867155    2565 scope.go:117] "RemoveContainer" containerID="54749e116a77c14baca460b83c2082d1c02ca879e9117b1e9a86b471114cf381"
Jul 28 21:42:10.868484 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.868463    2565 generic.go:356] "Generic (PLEG): container finished" podID="42444876-b7b4-44b7-b051-d70b44d906a3" containerID="b6e627c9b462a68c7b90db48dde03aeb68854a4c0cc6d2ea7a33f06b299f3f89" exitCode=0
Jul 28 21:42:10.868570 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.868530    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x" event={"ID":"42444876-b7b4-44b7-b051-d70b44d906a3","Type":"ContainerDied","Data":"b6e627c9b462a68c7b90db48dde03aeb68854a4c0cc6d2ea7a33f06b299f3f89"}
Jul 28 21:42:10.868570 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.868550    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x"
Jul 28 21:42:10.868682 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.868571    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x" event={"ID":"42444876-b7b4-44b7-b051-d70b44d906a3","Type":"ContainerDied","Data":"b27e4cac06c5fcbfdaebbbf050a23dd60de82847ecb8f6b2b62f0dc2ab7a4a1d"}
Jul 28 21:42:10.876484 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.876468    2565 scope.go:117] "RemoveContainer" containerID="54749e116a77c14baca460b83c2082d1c02ca879e9117b1e9a86b471114cf381"
Jul 28 21:42:10.876772 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:42:10.876752    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54749e116a77c14baca460b83c2082d1c02ca879e9117b1e9a86b471114cf381\": container with ID starting with 54749e116a77c14baca460b83c2082d1c02ca879e9117b1e9a86b471114cf381 not found: ID does not exist" containerID="54749e116a77c14baca460b83c2082d1c02ca879e9117b1e9a86b471114cf381"
Jul 28 21:42:10.876862 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.876785    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54749e116a77c14baca460b83c2082d1c02ca879e9117b1e9a86b471114cf381"} err="failed to get container status \"54749e116a77c14baca460b83c2082d1c02ca879e9117b1e9a86b471114cf381\": rpc error: code = NotFound desc = could not find container \"54749e116a77c14baca460b83c2082d1c02ca879e9117b1e9a86b471114cf381\": container with ID starting with 54749e116a77c14baca460b83c2082d1c02ca879e9117b1e9a86b471114cf381 not found: ID does not exist"
Jul 28 21:42:10.876862 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.876809    2565 scope.go:117] "RemoveContainer" containerID="b6e627c9b462a68c7b90db48dde03aeb68854a4c0cc6d2ea7a33f06b299f3f89"
Jul 28 21:42:10.885338 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.885310    2565 scope.go:117] "RemoveContainer" containerID="b6e627c9b462a68c7b90db48dde03aeb68854a4c0cc6d2ea7a33f06b299f3f89"
Jul 28 21:42:10.885574 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:42:10.885556    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e627c9b462a68c7b90db48dde03aeb68854a4c0cc6d2ea7a33f06b299f3f89\": container with ID starting with b6e627c9b462a68c7b90db48dde03aeb68854a4c0cc6d2ea7a33f06b299f3f89 not found: ID does not exist" containerID="b6e627c9b462a68c7b90db48dde03aeb68854a4c0cc6d2ea7a33f06b299f3f89"
Jul 28 21:42:10.885616 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.885581    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e627c9b462a68c7b90db48dde03aeb68854a4c0cc6d2ea7a33f06b299f3f89"} err="failed to get container status \"b6e627c9b462a68c7b90db48dde03aeb68854a4c0cc6d2ea7a33f06b299f3f89\": rpc error: code = NotFound desc = could not find container \"b6e627c9b462a68c7b90db48dde03aeb68854a4c0cc6d2ea7a33f06b299f3f89\": container with ID starting with b6e627c9b462a68c7b90db48dde03aeb68854a4c0cc6d2ea7a33f06b299f3f89 not found: ID does not exist"
Jul 28 21:42:10.899438 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.899417    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x"]
Jul 28 21:42:10.917183 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.917159    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d69596-848f64f98-f5q2x"]
Jul 28 21:42:10.935927 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.935904    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"]
Jul 28 21:42:10.944836 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:10.944813    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d69596-5776cb55d9-8wtrm"]
Jul 28 21:42:11.332340 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:11.332307    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c" path="/var/lib/kubelet/pods/11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c/volumes"
Jul 28 21:42:11.332805 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:11.332788    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42444876-b7b4-44b7-b051-d70b44d906a3" path="/var/lib/kubelet/pods/42444876-b7b4-44b7-b051-d70b44d906a3/volumes"
Jul 28 21:42:11.333094 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:11.333082    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2f1499a-16f1-4cb1-95ed-bcec74d0ca29" path="/var/lib/kubelet/pods/a2f1499a-16f1-4cb1-95ed-bcec74d0ca29/volumes"
Jul 28 21:42:12.866661 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:12.866592    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:42:16.832605 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:16.832568    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1096ca-vq8p7"]
Jul 28 21:42:16.833028 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:16.832960    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c" containerName="maas-api"
Jul 28 21:42:16.833028 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:16.832975    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c" containerName="maas-api"
Jul 28 21:42:16.833028 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:16.832997    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f1499a-16f1-4cb1-95ed-bcec74d0ca29" containerName="revoke-keys"
Jul 28 21:42:16.833028 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:16.833003    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f1499a-16f1-4cb1-95ed-bcec74d0ca29" containerName="revoke-keys"
Jul 28 21:42:16.833028 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:16.833011    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42444876-b7b4-44b7-b051-d70b44d906a3" containerName="payload-processing"
Jul 28 21:42:16.833028 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:16.833017    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="42444876-b7b4-44b7-b051-d70b44d906a3" containerName="payload-processing"
Jul 28 21:42:16.833217 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:16.833081    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="42444876-b7b4-44b7-b051-d70b44d906a3" containerName="payload-processing"
Jul 28 21:42:16.833217 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:16.833090    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="11dfd8a8-3adb-4e82-ac4d-1a23ee3f318c" containerName="maas-api"
Jul 28 21:42:16.833217 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:16.833098    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2f1499a-16f1-4cb1-95ed-bcec74d0ca29" containerName="revoke-keys"
Jul 28 21:42:16.837507 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:16.837491    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1096ca-vq8p7"
Jul 28 21:42:16.840157 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:16.840132    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w22xz\""
Jul 28 21:42:16.843103 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:16.843003    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1096ca-vq8p7"]
Jul 28 21:42:16.916196 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:16.916163    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5939f073-22c5-4bb7-90a9-bf4e361ffcdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-1096ca-vq8p7\" (UID: \"5939f073-22c5-4bb7-90a9-bf4e361ffcdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1096ca-vq8p7"
Jul 28 21:42:17.017545 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:17.017505    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5939f073-22c5-4bb7-90a9-bf4e361ffcdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-1096ca-vq8p7\" (UID: \"5939f073-22c5-4bb7-90a9-bf4e361ffcdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1096ca-vq8p7"
Jul 28 21:42:17.018189 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:17.018171    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5939f073-22c5-4bb7-90a9-bf4e361ffcdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-1096ca-vq8p7\" (UID: \"5939f073-22c5-4bb7-90a9-bf4e361ffcdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1096ca-vq8p7"
Jul 28 21:42:17.149079 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:17.148988    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1096ca-vq8p7"
Jul 28 21:42:17.287537 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:17.287501    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1096ca-vq8p7"]
Jul 28 21:42:17.289418 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:42:17.289392    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5939f073_22c5_4bb7_90a9_bf4e361ffcdf.slice/crio-cbd4a0a13fdb8c08b5cd5b7619b856cac91dcd23f961d86466518810a18a7ffe WatchSource:0}: Error finding container cbd4a0a13fdb8c08b5cd5b7619b856cac91dcd23f961d86466518810a18a7ffe: Status 404 returned error can't find the container with id cbd4a0a13fdb8c08b5cd5b7619b856cac91dcd23f961d86466518810a18a7ffe
Jul 28 21:42:17.912274 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:17.912240    2565 generic.go:356] "Generic (PLEG): container finished" podID="5939f073-22c5-4bb7-90a9-bf4e361ffcdf" containerID="763a9a2422ea13b125041385a1fa005bcd69e27d0b71e54fbdcc1d737d6f1afa" exitCode=0
Jul 28 21:42:17.912701 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:17.912334    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1096ca-vq8p7" event={"ID":"5939f073-22c5-4bb7-90a9-bf4e361ffcdf","Type":"ContainerDied","Data":"763a9a2422ea13b125041385a1fa005bcd69e27d0b71e54fbdcc1d737d6f1afa"}
Jul 28 21:42:17.912701 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:17.912400    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1096ca-vq8p7" event={"ID":"5939f073-22c5-4bb7-90a9-bf4e361ffcdf","Type":"ContainerStarted","Data":"cbd4a0a13fdb8c08b5cd5b7619b856cac91dcd23f961d86466518810a18a7ffe"}
Jul 28 21:42:19.050762 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:19.050739    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1096ca-vq8p7"
Jul 28 21:42:19.137322 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:19.137281    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5939f073-22c5-4bb7-90a9-bf4e361ffcdf-maas-api-service-ca\") pod \"5939f073-22c5-4bb7-90a9-bf4e361ffcdf\" (UID: \"5939f073-22c5-4bb7-90a9-bf4e361ffcdf\") "
Jul 28 21:42:19.137728 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:19.137704    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5939f073-22c5-4bb7-90a9-bf4e361ffcdf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5939f073-22c5-4bb7-90a9-bf4e361ffcdf" (UID: "5939f073-22c5-4bb7-90a9-bf4e361ffcdf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:42:19.238000 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:19.237964    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5939f073-22c5-4bb7-90a9-bf4e361ffcdf-maas-api-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:42:19.924272 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:19.924242    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1096ca-vq8p7"
Jul 28 21:42:19.924459 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:19.924287    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1096ca-vq8p7" event={"ID":"5939f073-22c5-4bb7-90a9-bf4e361ffcdf","Type":"ContainerDied","Data":"cbd4a0a13fdb8c08b5cd5b7619b856cac91dcd23f961d86466518810a18a7ffe"}
Jul 28 21:42:19.924459 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:19.924334    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbd4a0a13fdb8c08b5cd5b7619b856cac91dcd23f961d86466518810a18a7ffe"
Jul 28 21:42:27.389895 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.389859    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"]
Jul 28 21:42:27.390334 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.390066    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k" podUID="e258c9f3-c7eb-46a2-8c03-98bb9f2bda41" containerName="maas-api" containerID="cri-o://a6536e41e8c7d88802e68ff15a4c5b52055a16c9f809c83f8a87e5e8cf8846a1" gracePeriod=30
Jul 28 21:42:27.407853 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.407824    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1096ca-29754570-4rpx2"]
Jul 28 21:42:27.413031 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.413001    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1096ca-29754570-4rpx2"]
Jul 28 21:42:27.437355 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.437314    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2"]
Jul 28 21:42:27.437601 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.437548    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2" podUID="8a9ac520-5c17-4a81-a614-8e597ba74942" containerName="payload-processing" containerID="cri-o://70973c36453aa4cc1e232276a94d1b6f8f28db706965c2114624ba9a3ad2d07d" gracePeriod=30
Jul 28 21:42:27.445819 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.445792    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b"]
Jul 28 21:42:27.446047 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.446022    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b" podUID="c60268ae-e839-4cb8-8b7c-422893332578" containerName="payload-pre-processing" containerID="cri-o://2ada3a5faa22b4ebae4b004bff65c16dc8c0d8e1d140678561d4d0ec9e38c0db" gracePeriod=30
Jul 28 21:42:27.577805 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.577744    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1096ca-vq8p7"]
Jul 28 21:42:27.580545 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.580511    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-1096ca-vq8p7"]
Jul 28 21:42:27.702995 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.702971    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"
Jul 28 21:42:27.735739 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.735714    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b"
Jul 28 21:42:27.740130 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.740106    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2"
Jul 28 21:42:27.813003 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.812970    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9slr\" (UniqueName: \"kubernetes.io/projected/8a9ac520-5c17-4a81-a614-8e597ba74942-kube-api-access-h9slr\") pod \"8a9ac520-5c17-4a81-a614-8e597ba74942\" (UID: \"8a9ac520-5c17-4a81-a614-8e597ba74942\") "
Jul 28 21:42:27.813155 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.813049    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhb9w\" (UniqueName: \"kubernetes.io/projected/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-kube-api-access-hhb9w\") pod \"e258c9f3-c7eb-46a2-8c03-98bb9f2bda41\" (UID: \"e258c9f3-c7eb-46a2-8c03-98bb9f2bda41\") "
Jul 28 21:42:27.813155 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.813144    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9ac520-5c17-4a81-a614-8e597ba74942-plugins-config-volume\") pod \"8a9ac520-5c17-4a81-a614-8e597ba74942\" (UID: \"8a9ac520-5c17-4a81-a614-8e597ba74942\") "
Jul 28 21:42:27.813250 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.813192    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-maas-api-tls\") pod \"e258c9f3-c7eb-46a2-8c03-98bb9f2bda41\" (UID: \"e258c9f3-c7eb-46a2-8c03-98bb9f2bda41\") "
Jul 28 21:42:27.813250 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.813220    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqvh2\" (UniqueName: \"kubernetes.io/projected/c60268ae-e839-4cb8-8b7c-422893332578-kube-api-access-fqvh2\") pod \"c60268ae-e839-4cb8-8b7c-422893332578\" (UID: \"c60268ae-e839-4cb8-8b7c-422893332578\") "
Jul 28 21:42:27.813338 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.813245    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c60268ae-e839-4cb8-8b7c-422893332578-plugins-config-volume\") pod \"c60268ae-e839-4cb8-8b7c-422893332578\" (UID: \"c60268ae-e839-4cb8-8b7c-422893332578\") "
Jul 28 21:42:27.813338 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.813283    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-openshift-service-ca\") pod \"e258c9f3-c7eb-46a2-8c03-98bb9f2bda41\" (UID: \"e258c9f3-c7eb-46a2-8c03-98bb9f2bda41\") "
Jul 28 21:42:27.813942 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.813589    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9ac520-5c17-4a81-a614-8e597ba74942-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8a9ac520-5c17-4a81-a614-8e597ba74942" (UID: "8a9ac520-5c17-4a81-a614-8e597ba74942"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:42:27.813942 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.813706    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c60268ae-e839-4cb8-8b7c-422893332578-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c60268ae-e839-4cb8-8b7c-422893332578" (UID: "c60268ae-e839-4cb8-8b7c-422893332578"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:42:27.813942 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.813885    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e258c9f3-c7eb-46a2-8c03-98bb9f2bda41" (UID: "e258c9f3-c7eb-46a2-8c03-98bb9f2bda41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:42:27.815443 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.815420    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e258c9f3-c7eb-46a2-8c03-98bb9f2bda41" (UID: "e258c9f3-c7eb-46a2-8c03-98bb9f2bda41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:42:27.815542 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.815455    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60268ae-e839-4cb8-8b7c-422893332578-kube-api-access-fqvh2" (OuterVolumeSpecName: "kube-api-access-fqvh2") pod "c60268ae-e839-4cb8-8b7c-422893332578" (UID: "c60268ae-e839-4cb8-8b7c-422893332578"). InnerVolumeSpecName "kube-api-access-fqvh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:42:27.815542 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.815487    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-kube-api-access-hhb9w" (OuterVolumeSpecName: "kube-api-access-hhb9w") pod "e258c9f3-c7eb-46a2-8c03-98bb9f2bda41" (UID: "e258c9f3-c7eb-46a2-8c03-98bb9f2bda41"). InnerVolumeSpecName "kube-api-access-hhb9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:42:27.815618 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.815562    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9ac520-5c17-4a81-a614-8e597ba74942-kube-api-access-h9slr" (OuterVolumeSpecName: "kube-api-access-h9slr") pod "8a9ac520-5c17-4a81-a614-8e597ba74942" (UID: "8a9ac520-5c17-4a81-a614-8e597ba74942"). InnerVolumeSpecName "kube-api-access-h9slr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:42:27.914605 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.914566    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c60268ae-e839-4cb8-8b7c-422893332578-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:42:27.914605 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.914595    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-openshift-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:42:27.914605 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.914605    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9slr\" (UniqueName: \"kubernetes.io/projected/8a9ac520-5c17-4a81-a614-8e597ba74942-kube-api-access-h9slr\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:42:27.914605 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.914614    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhb9w\" (UniqueName: \"kubernetes.io/projected/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-kube-api-access-hhb9w\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:42:27.914933 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.914622    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9ac520-5c17-4a81-a614-8e597ba74942-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:42:27.914933 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.914655    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41-maas-api-tls\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:42:27.914933 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.914664    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqvh2\" (UniqueName: \"kubernetes.io/projected/c60268ae-e839-4cb8-8b7c-422893332578-kube-api-access-fqvh2\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:42:27.971709 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.971676    2565 generic.go:356] "Generic (PLEG): container finished" podID="e258c9f3-c7eb-46a2-8c03-98bb9f2bda41" containerID="a6536e41e8c7d88802e68ff15a4c5b52055a16c9f809c83f8a87e5e8cf8846a1" exitCode=0
Jul 28 21:42:27.971901 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.971726    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"
Jul 28 21:42:27.971901 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.971763    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k" event={"ID":"e258c9f3-c7eb-46a2-8c03-98bb9f2bda41","Type":"ContainerDied","Data":"a6536e41e8c7d88802e68ff15a4c5b52055a16c9f809c83f8a87e5e8cf8846a1"}
Jul 28 21:42:27.971901 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.971817    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k" event={"ID":"e258c9f3-c7eb-46a2-8c03-98bb9f2bda41","Type":"ContainerDied","Data":"7c89ed9d2c772473729270214e5535a85571d1d49879d80d04541f0c7051e110"}
Jul 28 21:42:27.971901 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.971849    2565 scope.go:117] "RemoveContainer" containerID="a6536e41e8c7d88802e68ff15a4c5b52055a16c9f809c83f8a87e5e8cf8846a1"
Jul 28 21:42:27.973037 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.973013    2565 generic.go:356] "Generic (PLEG): container finished" podID="8a9ac520-5c17-4a81-a614-8e597ba74942" containerID="70973c36453aa4cc1e232276a94d1b6f8f28db706965c2114624ba9a3ad2d07d" exitCode=0
Jul 28 21:42:27.973097 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.973055    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2"
Jul 28 21:42:27.973152 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.973108    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2" event={"ID":"8a9ac520-5c17-4a81-a614-8e597ba74942","Type":"ContainerDied","Data":"70973c36453aa4cc1e232276a94d1b6f8f28db706965c2114624ba9a3ad2d07d"}
Jul 28 21:42:27.973211 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.973165    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2" event={"ID":"8a9ac520-5c17-4a81-a614-8e597ba74942","Type":"ContainerDied","Data":"10a98d3fa14f3f33e589af889a8fd63c6755d5bb02aeba4ea83ffa1f26c93562"}
Jul 28 21:42:27.974649 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.974617    2565 generic.go:356] "Generic (PLEG): container finished" podID="c60268ae-e839-4cb8-8b7c-422893332578" containerID="2ada3a5faa22b4ebae4b004bff65c16dc8c0d8e1d140678561d4d0ec9e38c0db" exitCode=0
Jul 28 21:42:27.974815 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.974659    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b" event={"ID":"c60268ae-e839-4cb8-8b7c-422893332578","Type":"ContainerDied","Data":"2ada3a5faa22b4ebae4b004bff65c16dc8c0d8e1d140678561d4d0ec9e38c0db"}
Jul 28 21:42:27.974925 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.974846    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b" event={"ID":"c60268ae-e839-4cb8-8b7c-422893332578","Type":"ContainerDied","Data":"ca76d0e5350ea0266a7d0fe348ed7bf48e6158af260c2810d62045f1def66f4d"}
Jul 28 21:42:27.974925 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.974689    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b"
Jul 28 21:42:27.983060 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.983044    2565 scope.go:117] "RemoveContainer" containerID="a6536e41e8c7d88802e68ff15a4c5b52055a16c9f809c83f8a87e5e8cf8846a1"
Jul 28 21:42:27.983326 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:42:27.983305    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6536e41e8c7d88802e68ff15a4c5b52055a16c9f809c83f8a87e5e8cf8846a1\": container with ID starting with a6536e41e8c7d88802e68ff15a4c5b52055a16c9f809c83f8a87e5e8cf8846a1 not found: ID does not exist" containerID="a6536e41e8c7d88802e68ff15a4c5b52055a16c9f809c83f8a87e5e8cf8846a1"
Jul 28 21:42:27.983377 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.983334    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6536e41e8c7d88802e68ff15a4c5b52055a16c9f809c83f8a87e5e8cf8846a1"} err="failed to get container status \"a6536e41e8c7d88802e68ff15a4c5b52055a16c9f809c83f8a87e5e8cf8846a1\": rpc error: code = NotFound desc = could not find container \"a6536e41e8c7d88802e68ff15a4c5b52055a16c9f809c83f8a87e5e8cf8846a1\": container with ID starting with a6536e41e8c7d88802e68ff15a4c5b52055a16c9f809c83f8a87e5e8cf8846a1 not found: ID does not exist"
Jul 28 21:42:27.983377 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.983350    2565 scope.go:117] "RemoveContainer" containerID="70973c36453aa4cc1e232276a94d1b6f8f28db706965c2114624ba9a3ad2d07d"
Jul 28 21:42:27.991891 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.991871    2565 scope.go:117] "RemoveContainer" containerID="70973c36453aa4cc1e232276a94d1b6f8f28db706965c2114624ba9a3ad2d07d"
Jul 28 21:42:27.992140 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:42:27.992120    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70973c36453aa4cc1e232276a94d1b6f8f28db706965c2114624ba9a3ad2d07d\": container with ID starting with 70973c36453aa4cc1e232276a94d1b6f8f28db706965c2114624ba9a3ad2d07d not found: ID does not exist" containerID="70973c36453aa4cc1e232276a94d1b6f8f28db706965c2114624ba9a3ad2d07d"
Jul 28 21:42:27.992181 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.992149    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70973c36453aa4cc1e232276a94d1b6f8f28db706965c2114624ba9a3ad2d07d"} err="failed to get container status \"70973c36453aa4cc1e232276a94d1b6f8f28db706965c2114624ba9a3ad2d07d\": rpc error: code = NotFound desc = could not find container \"70973c36453aa4cc1e232276a94d1b6f8f28db706965c2114624ba9a3ad2d07d\": container with ID starting with 70973c36453aa4cc1e232276a94d1b6f8f28db706965c2114624ba9a3ad2d07d not found: ID does not exist"
Jul 28 21:42:27.992181 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.992167    2565 scope.go:117] "RemoveContainer" containerID="2ada3a5faa22b4ebae4b004bff65c16dc8c0d8e1d140678561d4d0ec9e38c0db"
Jul 28 21:42:27.995570 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.995549    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"]
Jul 28 21:42:27.997931 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:27.997908    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1096ca-59789fb68c-8m56k"]
Jul 28 21:42:28.001014 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:28.000998    2565 scope.go:117] "RemoveContainer" containerID="2ada3a5faa22b4ebae4b004bff65c16dc8c0d8e1d140678561d4d0ec9e38c0db"
Jul 28 21:42:28.001258 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:42:28.001241    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ada3a5faa22b4ebae4b004bff65c16dc8c0d8e1d140678561d4d0ec9e38c0db\": container with ID starting with 2ada3a5faa22b4ebae4b004bff65c16dc8c0d8e1d140678561d4d0ec9e38c0db not found: ID does not exist" containerID="2ada3a5faa22b4ebae4b004bff65c16dc8c0d8e1d140678561d4d0ec9e38c0db"
Jul 28 21:42:28.001315 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:28.001266    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ada3a5faa22b4ebae4b004bff65c16dc8c0d8e1d140678561d4d0ec9e38c0db"} err="failed to get container status \"2ada3a5faa22b4ebae4b004bff65c16dc8c0d8e1d140678561d4d0ec9e38c0db\": rpc error: code = NotFound desc = could not find container \"2ada3a5faa22b4ebae4b004bff65c16dc8c0d8e1d140678561d4d0ec9e38c0db\": container with ID starting with 2ada3a5faa22b4ebae4b004bff65c16dc8c0d8e1d140678561d4d0ec9e38c0db not found: ID does not exist"
Jul 28 21:42:28.007546 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:28.007522    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b"]
Jul 28 21:42:28.011536 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:28.011517    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1096ca-7f9b7f6648-fhz8b"]
Jul 28 21:42:28.020098 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:28.020079    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2"]
Jul 28 21:42:28.023603 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:28.023585    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1096ca-7d98c94bbb-q5ck2"]
Jul 28 21:42:29.330759 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:29.330729    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5939f073-22c5-4bb7-90a9-bf4e361ffcdf" path="/var/lib/kubelet/pods/5939f073-22c5-4bb7-90a9-bf4e361ffcdf/volumes"
Jul 28 21:42:29.331125 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:29.330991    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86108fef-e226-42c7-99f9-0b544415f8f0" path="/var/lib/kubelet/pods/86108fef-e226-42c7-99f9-0b544415f8f0/volumes"
Jul 28 21:42:29.331259 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:29.331247    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a9ac520-5c17-4a81-a614-8e597ba74942" path="/var/lib/kubelet/pods/8a9ac520-5c17-4a81-a614-8e597ba74942/volumes"
Jul 28 21:42:29.331552 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:29.331541    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c60268ae-e839-4cb8-8b7c-422893332578" path="/var/lib/kubelet/pods/c60268ae-e839-4cb8-8b7c-422893332578/volumes"
Jul 28 21:42:29.332196 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:29.332182    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e258c9f3-c7eb-46a2-8c03-98bb9f2bda41" path="/var/lib/kubelet/pods/e258c9f3-c7eb-46a2-8c03-98bb9f2bda41/volumes"
Jul 28 21:42:30.454894 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:30.454859    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:42:34.634379 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:34.634349    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2c3f70-klhsn"]
Jul 28 21:42:34.634798 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:34.634758    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c60268ae-e839-4cb8-8b7c-422893332578" containerName="payload-pre-processing"
Jul 28 21:42:34.634798 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:34.634770    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c60268ae-e839-4cb8-8b7c-422893332578" containerName="payload-pre-processing"
Jul 28 21:42:34.634798 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:34.634786    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e258c9f3-c7eb-46a2-8c03-98bb9f2bda41" containerName="maas-api"
Jul 28 21:42:34.634798 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:34.634792    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e258c9f3-c7eb-46a2-8c03-98bb9f2bda41" containerName="maas-api"
Jul 28 21:42:34.634798 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:34.634799    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5939f073-22c5-4bb7-90a9-bf4e361ffcdf" containerName="revoke-keys"
Jul 28 21:42:34.634967 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:34.634806    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5939f073-22c5-4bb7-90a9-bf4e361ffcdf" containerName="revoke-keys"
Jul 28 21:42:34.634967 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:34.634814    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9ac520-5c17-4a81-a614-8e597ba74942" containerName="payload-processing"
Jul 28 21:42:34.634967 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:34.634819    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9ac520-5c17-4a81-a614-8e597ba74942" containerName="payload-processing"
Jul 28 21:42:34.634967 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:34.634873    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e258c9f3-c7eb-46a2-8c03-98bb9f2bda41" containerName="maas-api"
Jul 28 21:42:34.634967 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:34.634882    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a9ac520-5c17-4a81-a614-8e597ba74942" containerName="payload-processing"
Jul 28 21:42:34.634967 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:34.634889    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c60268ae-e839-4cb8-8b7c-422893332578" containerName="payload-pre-processing"
Jul 28 21:42:34.634967 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:34.634897    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5939f073-22c5-4bb7-90a9-bf4e361ffcdf" containerName="revoke-keys"
Jul 28 21:42:34.639560 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:34.639542    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2c3f70-klhsn"
Jul 28 21:42:34.641645 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:34.641619    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w22xz\""
Jul 28 21:42:34.647999 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:34.647971    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2c3f70-klhsn"]
Jul 28 21:42:34.774315 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:34.774279    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfb1de7-d83f-4868-8b08-fa19d7764811-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-2c3f70-klhsn\" (UID: \"3cfb1de7-d83f-4868-8b08-fa19d7764811\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2c3f70-klhsn"
Jul 28 21:42:34.875707 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:34.875676    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfb1de7-d83f-4868-8b08-fa19d7764811-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-2c3f70-klhsn\" (UID: \"3cfb1de7-d83f-4868-8b08-fa19d7764811\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2c3f70-klhsn"
Jul 28 21:42:34.876313 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:34.876280    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfb1de7-d83f-4868-8b08-fa19d7764811-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-2c3f70-klhsn\" (UID: \"3cfb1de7-d83f-4868-8b08-fa19d7764811\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2c3f70-klhsn"
Jul 28 21:42:34.951752 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:34.951720    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2c3f70-klhsn"
Jul 28 21:42:35.093514 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:35.093486    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2c3f70-klhsn"]
Jul 28 21:42:35.095456 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:42:35.095431    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cfb1de7_d83f_4868_8b08_fa19d7764811.slice/crio-325f8eac1ffd2b7f34cc0b0588f11ad14ba97d35700e98af9aaed58c8d7e86cb WatchSource:0}: Error finding container 325f8eac1ffd2b7f34cc0b0588f11ad14ba97d35700e98af9aaed58c8d7e86cb: Status 404 returned error can't find the container with id 325f8eac1ffd2b7f34cc0b0588f11ad14ba97d35700e98af9aaed58c8d7e86cb
Jul 28 21:42:36.027107 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:36.027009    2565 generic.go:356] "Generic (PLEG): container finished" podID="3cfb1de7-d83f-4868-8b08-fa19d7764811" containerID="f66912445d5d1517da654c97f0ee8c7ecdce528f032eb7e01c38f664f354396d" exitCode=0
Jul 28 21:42:36.027495 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:36.027093    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2c3f70-klhsn" event={"ID":"3cfb1de7-d83f-4868-8b08-fa19d7764811","Type":"ContainerDied","Data":"f66912445d5d1517da654c97f0ee8c7ecdce528f032eb7e01c38f664f354396d"}
Jul 28 21:42:36.027495 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:36.027155    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2c3f70-klhsn" event={"ID":"3cfb1de7-d83f-4868-8b08-fa19d7764811","Type":"ContainerStarted","Data":"325f8eac1ffd2b7f34cc0b0588f11ad14ba97d35700e98af9aaed58c8d7e86cb"}
Jul 28 21:42:37.174294 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:37.174267    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2c3f70-klhsn"
Jul 28 21:42:37.303190 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:37.303092    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfb1de7-d83f-4868-8b08-fa19d7764811-maas-api-service-ca\") pod \"3cfb1de7-d83f-4868-8b08-fa19d7764811\" (UID: \"3cfb1de7-d83f-4868-8b08-fa19d7764811\") "
Jul 28 21:42:37.303490 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:37.303464    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cfb1de7-d83f-4868-8b08-fa19d7764811-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3cfb1de7-d83f-4868-8b08-fa19d7764811" (UID: "3cfb1de7-d83f-4868-8b08-fa19d7764811"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:42:37.404416 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:37.404373    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfb1de7-d83f-4868-8b08-fa19d7764811-maas-api-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:42:38.040925 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:38.040891    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2c3f70-klhsn"
Jul 28 21:42:38.041112 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:38.040929    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2c3f70-klhsn" event={"ID":"3cfb1de7-d83f-4868-8b08-fa19d7764811","Type":"ContainerDied","Data":"325f8eac1ffd2b7f34cc0b0588f11ad14ba97d35700e98af9aaed58c8d7e86cb"}
Jul 28 21:42:38.041112 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:38.040976    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="325f8eac1ffd2b7f34cc0b0588f11ad14ba97d35700e98af9aaed58c8d7e86cb"
Jul 28 21:42:45.219145 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.219109    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"]
Jul 28 21:42:45.219627 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.219360    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf" podUID="af83bbaf-5051-4c55-8d86-d6b7dc35790d" containerName="maas-api" containerID="cri-o://3a93bd682b5429d99b8606b8bd9df1482975478cca2b35c505bff9882c887230" gracePeriod=30
Jul 28 21:42:45.253235 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.253199    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2c3f70-29754570-xbx5x"]
Jul 28 21:42:45.256408 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.256380    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2c3f70-29754570-xbx5x"]
Jul 28 21:42:45.287570 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.287535    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp"]
Jul 28 21:42:45.287856 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.287826    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp" podUID="a43b4211-355b-4736-a310-712b4761f4db" containerName="payload-pre-processing" containerID="cri-o://22d6faad98194f9765192cf614c9189a82ee46df4aadfb94f389cc4ff6be696d" gracePeriod=30
Jul 28 21:42:45.333182 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.333149    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60cc0afc-b1b3-4d76-9f61-1a5a270daea9" path="/var/lib/kubelet/pods/60cc0afc-b1b3-4d76-9f61-1a5a270daea9/volumes"
Jul 28 21:42:45.412730 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.412689    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2c3f70-klhsn"]
Jul 28 21:42:45.414958 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.414931    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2c3f70-klhsn"]
Jul 28 21:42:45.557532 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.557512    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"
Jul 28 21:42:45.560398 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.560374    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp"
Jul 28 21:42:45.679906 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.679866    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af83bbaf-5051-4c55-8d86-d6b7dc35790d-maas-api-tls\") pod \"af83bbaf-5051-4c55-8d86-d6b7dc35790d\" (UID: \"af83bbaf-5051-4c55-8d86-d6b7dc35790d\") "
Jul 28 21:42:45.680089 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.679914    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af83bbaf-5051-4c55-8d86-d6b7dc35790d-openshift-service-ca\") pod \"af83bbaf-5051-4c55-8d86-d6b7dc35790d\" (UID: \"af83bbaf-5051-4c55-8d86-d6b7dc35790d\") "
Jul 28 21:42:45.680089 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.679946    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsz7t\" (UniqueName: \"kubernetes.io/projected/af83bbaf-5051-4c55-8d86-d6b7dc35790d-kube-api-access-bsz7t\") pod \"af83bbaf-5051-4c55-8d86-d6b7dc35790d\" (UID: \"af83bbaf-5051-4c55-8d86-d6b7dc35790d\") "
Jul 28 21:42:45.680089 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.679967    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-722gq\" (UniqueName: \"kubernetes.io/projected/a43b4211-355b-4736-a310-712b4761f4db-kube-api-access-722gq\") pod \"a43b4211-355b-4736-a310-712b4761f4db\" (UID: \"a43b4211-355b-4736-a310-712b4761f4db\") "
Jul 28 21:42:45.680089 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.680007    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43b4211-355b-4736-a310-712b4761f4db-plugins-config-volume\") pod \"a43b4211-355b-4736-a310-712b4761f4db\" (UID: \"a43b4211-355b-4736-a310-712b4761f4db\") "
Jul 28 21:42:45.680395 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.680363    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af83bbaf-5051-4c55-8d86-d6b7dc35790d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "af83bbaf-5051-4c55-8d86-d6b7dc35790d" (UID: "af83bbaf-5051-4c55-8d86-d6b7dc35790d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:42:45.680523 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.680415    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a43b4211-355b-4736-a310-712b4761f4db-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a43b4211-355b-4736-a310-712b4761f4db" (UID: "a43b4211-355b-4736-a310-712b4761f4db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 21:42:45.682238 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.682210    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43b4211-355b-4736-a310-712b4761f4db-kube-api-access-722gq" (OuterVolumeSpecName: "kube-api-access-722gq") pod "a43b4211-355b-4736-a310-712b4761f4db" (UID: "a43b4211-355b-4736-a310-712b4761f4db"). InnerVolumeSpecName "kube-api-access-722gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:42:45.682583 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.682556    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af83bbaf-5051-4c55-8d86-d6b7dc35790d-kube-api-access-bsz7t" (OuterVolumeSpecName: "kube-api-access-bsz7t") pod "af83bbaf-5051-4c55-8d86-d6b7dc35790d" (UID: "af83bbaf-5051-4c55-8d86-d6b7dc35790d"). InnerVolumeSpecName "kube-api-access-bsz7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 21:42:45.682583 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.682562    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af83bbaf-5051-4c55-8d86-d6b7dc35790d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "af83bbaf-5051-4c55-8d86-d6b7dc35790d" (UID: "af83bbaf-5051-4c55-8d86-d6b7dc35790d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 21:42:45.781202 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.781098    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af83bbaf-5051-4c55-8d86-d6b7dc35790d-maas-api-tls\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:42:45.781202 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.781141    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af83bbaf-5051-4c55-8d86-d6b7dc35790d-openshift-service-ca\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:42:45.781202 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.781153    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsz7t\" (UniqueName: \"kubernetes.io/projected/af83bbaf-5051-4c55-8d86-d6b7dc35790d-kube-api-access-bsz7t\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:42:45.781202 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.781162    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-722gq\" (UniqueName: \"kubernetes.io/projected/a43b4211-355b-4736-a310-712b4761f4db-kube-api-access-722gq\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:42:45.781202 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:45.781171    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43b4211-355b-4736-a310-712b4761f4db-plugins-config-volume\") on node \"ip-10-0-134-170.ec2.internal\" DevicePath \"\""
Jul 28 21:42:46.093825 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:46.093723    2565 generic.go:356] "Generic (PLEG): container finished" podID="af83bbaf-5051-4c55-8d86-d6b7dc35790d" containerID="3a93bd682b5429d99b8606b8bd9df1482975478cca2b35c505bff9882c887230" exitCode=0
Jul 28 21:42:46.093825 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:46.093788    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"
Jul 28 21:42:46.093825 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:46.093811    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf" event={"ID":"af83bbaf-5051-4c55-8d86-d6b7dc35790d","Type":"ContainerDied","Data":"3a93bd682b5429d99b8606b8bd9df1482975478cca2b35c505bff9882c887230"}
Jul 28 21:42:46.094141 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:46.093914    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf" event={"ID":"af83bbaf-5051-4c55-8d86-d6b7dc35790d","Type":"ContainerDied","Data":"05b1463535b8f130543b47edcf38a42c9a3baad36bf586b244c2803340b1d9c2"}
Jul 28 21:42:46.094141 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:46.093945    2565 scope.go:117] "RemoveContainer" containerID="3a93bd682b5429d99b8606b8bd9df1482975478cca2b35c505bff9882c887230"
Jul 28 21:42:46.095272 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:46.095249    2565 generic.go:356] "Generic (PLEG): container finished" podID="a43b4211-355b-4736-a310-712b4761f4db" containerID="22d6faad98194f9765192cf614c9189a82ee46df4aadfb94f389cc4ff6be696d" exitCode=0
Jul 28 21:42:46.095385 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:46.095303    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp" event={"ID":"a43b4211-355b-4736-a310-712b4761f4db","Type":"ContainerDied","Data":"22d6faad98194f9765192cf614c9189a82ee46df4aadfb94f389cc4ff6be696d"}
Jul 28 21:42:46.095385 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:46.095312    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp"
Jul 28 21:42:46.095385 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:46.095344    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp" event={"ID":"a43b4211-355b-4736-a310-712b4761f4db","Type":"ContainerDied","Data":"c2b7bdf1894968937daa7faac9bbd58cdc34e460c70e43914b8f12c3a6236bf9"}
Jul 28 21:42:46.103097 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:46.103080    2565 scope.go:117] "RemoveContainer" containerID="3a93bd682b5429d99b8606b8bd9df1482975478cca2b35c505bff9882c887230"
Jul 28 21:42:46.103408 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:42:46.103384    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a93bd682b5429d99b8606b8bd9df1482975478cca2b35c505bff9882c887230\": container with ID starting with 3a93bd682b5429d99b8606b8bd9df1482975478cca2b35c505bff9882c887230 not found: ID does not exist" containerID="3a93bd682b5429d99b8606b8bd9df1482975478cca2b35c505bff9882c887230"
Jul 28 21:42:46.103497 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:46.103413    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a93bd682b5429d99b8606b8bd9df1482975478cca2b35c505bff9882c887230"} err="failed to get container status \"3a93bd682b5429d99b8606b8bd9df1482975478cca2b35c505bff9882c887230\": rpc error: code = NotFound desc = could not find container \"3a93bd682b5429d99b8606b8bd9df1482975478cca2b35c505bff9882c887230\": container with ID starting with 3a93bd682b5429d99b8606b8bd9df1482975478cca2b35c505bff9882c887230 not found: ID does not exist"
Jul 28 21:42:46.103497 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:46.103428    2565 scope.go:117] "RemoveContainer" containerID="22d6faad98194f9765192cf614c9189a82ee46df4aadfb94f389cc4ff6be696d"
Jul 28 21:42:46.112442 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:46.112424    2565 scope.go:117] "RemoveContainer" containerID="22d6faad98194f9765192cf614c9189a82ee46df4aadfb94f389cc4ff6be696d"
Jul 28 21:42:46.112715 ip-10-0-134-170 kubenswrapper[2565]: E0728 21:42:46.112694    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d6faad98194f9765192cf614c9189a82ee46df4aadfb94f389cc4ff6be696d\": container with ID starting with 22d6faad98194f9765192cf614c9189a82ee46df4aadfb94f389cc4ff6be696d not found: ID does not exist" containerID="22d6faad98194f9765192cf614c9189a82ee46df4aadfb94f389cc4ff6be696d"
Jul 28 21:42:46.112761 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:46.112723    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d6faad98194f9765192cf614c9189a82ee46df4aadfb94f389cc4ff6be696d"} err="failed to get container status \"22d6faad98194f9765192cf614c9189a82ee46df4aadfb94f389cc4ff6be696d\": rpc error: code = NotFound desc = could not find container \"22d6faad98194f9765192cf614c9189a82ee46df4aadfb94f389cc4ff6be696d\": container with ID starting with 22d6faad98194f9765192cf614c9189a82ee46df4aadfb94f389cc4ff6be696d not found: ID does not exist"
Jul 28 21:42:46.122953 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:46.122917    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"]
Jul 28 21:42:46.127877 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:46.127852    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2c3f70-68467b584c-7k9bf"]
Jul 28 21:42:46.141004 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:46.140971    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp"]
Jul 28 21:42:46.143192 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:46.143171    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-2c3f70-6dc8d7c9b8-lngtp"]
Jul 28 21:42:47.331375 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:47.331295    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cfb1de7-d83f-4868-8b08-fa19d7764811" path="/var/lib/kubelet/pods/3cfb1de7-d83f-4868-8b08-fa19d7764811/volumes"
Jul 28 21:42:47.331787 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:47.331778    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a43b4211-355b-4736-a310-712b4761f4db" path="/var/lib/kubelet/pods/a43b4211-355b-4736-a310-712b4761f4db/volumes"
Jul 28 21:42:47.332126 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:47.332112    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af83bbaf-5051-4c55-8d86-d6b7dc35790d" path="/var/lib/kubelet/pods/af83bbaf-5051-4c55-8d86-d6b7dc35790d/volumes"
Jul 28 21:42:48.171668 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:48.171614    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nbrxn"]
Jul 28 21:42:58.434040 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:58.434010    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mdl2q_c1831985-93ba-4383-bcb3-08079db37381/manager/0.log"
Jul 28 21:42:58.549926 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:58.549894    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cc774576c-57mmd_b2545b75-760e-498e-b20c-f417090b67d0/copy-manifests/0.log"
Jul 28 21:42:58.556458 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:58.556437    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cc774576c-57mmd_b2545b75-760e-498e-b20c-f417090b67d0/manager/0.log"
Jul 28 21:42:59.247439 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:59.247403    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754540-q2htm_d5cb520e-0e96-4ca8-893a-63cbcbda2ab8/cleanup/0.log"
Jul 28 21:42:59.358011 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:59.357971    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754555-2jgkr_a76793ab-4208-4f93-852b-578e523df8b1/cleanup/0.log"
Jul 28 21:42:59.463888 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:42:59.463855    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754570-2txlk_8cc3fd82-4f33-4edf-b053-563b1ef08cae/cleanup/0.log"
Jul 28 21:43:00.328146 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:00.328106    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j_51aa2b30-ff05-45fa-90db-5aa6bf4970ae/util/0.log"
Jul 28 21:43:00.334555 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:00.334526    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j_51aa2b30-ff05-45fa-90db-5aa6bf4970ae/pull/0.log"
Jul 28 21:43:00.340028 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:00.340011    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j_51aa2b30-ff05-45fa-90db-5aa6bf4970ae/extract/0.log"
Jul 28 21:43:00.450758 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:00.450722    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz_7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4/util/0.log"
Jul 28 21:43:00.457049 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:00.457033    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz_7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4/pull/0.log"
Jul 28 21:43:00.462552 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:00.462532    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz_7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4/extract/0.log"
Jul 28 21:43:00.572960 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:00.572921    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn_073b3df5-255a-43d8-bdf8-1e75b767507a/util/0.log"
Jul 28 21:43:00.579073 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:00.579013    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn_073b3df5-255a-43d8-bdf8-1e75b767507a/pull/0.log"
Jul 28 21:43:00.584597 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:00.584578    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn_073b3df5-255a-43d8-bdf8-1e75b767507a/extract/0.log"
Jul 28 21:43:00.696147 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:00.696120    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn_ed38cb3e-13af-4f80-b366-065ffe40d3f4/util/0.log"
Jul 28 21:43:00.702880 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:00.702858    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn_ed38cb3e-13af-4f80-b366-065ffe40d3f4/pull/0.log"
Jul 28 21:43:00.714646 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:00.714601    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn_ed38cb3e-13af-4f80-b366-065ffe40d3f4/extract/0.log"
Jul 28 21:43:01.152903 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:01.152875    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-6zq5m_f94621ee-e112-4507-b648-81c49a3d370b/kuadrant-console-plugin/0.log"
Jul 28 21:43:01.496371 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:01.496339    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-nbrxn_37a1bb31-c8f7-465e-8610-e7ae408a4a74/limitador/0.log"
Jul 28 21:43:02.168723 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:02.168697    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68cb4f9b6-gxtld_19df36da-d263-4e8f-8589-45e88f835635/kube-auth-proxy/0.log"
Jul 28 21:43:08.631961 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.631921    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-grfnp/must-gather-58jrl"]
Jul 28 21:43:08.632368 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.632309    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af83bbaf-5051-4c55-8d86-d6b7dc35790d" containerName="maas-api"
Jul 28 21:43:08.632368 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.632323    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="af83bbaf-5051-4c55-8d86-d6b7dc35790d" containerName="maas-api"
Jul 28 21:43:08.632368 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.632330    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cfb1de7-d83f-4868-8b08-fa19d7764811" containerName="revoke-keys"
Jul 28 21:43:08.632368 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.632336    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cfb1de7-d83f-4868-8b08-fa19d7764811" containerName="revoke-keys"
Jul 28 21:43:08.632368 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.632346    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a43b4211-355b-4736-a310-712b4761f4db" containerName="payload-pre-processing"
Jul 28 21:43:08.632368 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.632355    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a43b4211-355b-4736-a310-712b4761f4db" containerName="payload-pre-processing"
Jul 28 21:43:08.632579 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.632419    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="af83bbaf-5051-4c55-8d86-d6b7dc35790d" containerName="maas-api"
Jul 28 21:43:08.632579 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.632431    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cfb1de7-d83f-4868-8b08-fa19d7764811" containerName="revoke-keys"
Jul 28 21:43:08.632579 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.632442    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a43b4211-355b-4736-a310-712b4761f4db" containerName="payload-pre-processing"
Jul 28 21:43:08.635627 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.635611    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grfnp/must-gather-58jrl"
Jul 28 21:43:08.637380 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.637352    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-grfnp\"/\"default-dockercfg-trz77\""
Jul 28 21:43:08.637500 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.637451    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-grfnp\"/\"openshift-service-ca.crt\""
Jul 28 21:43:08.637598 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.637580    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-grfnp\"/\"kube-root-ca.crt\""
Jul 28 21:43:08.646588 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.646561    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grfnp/must-gather-58jrl"]
Jul 28 21:43:08.658264 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.658231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs4cr\" (UniqueName: \"kubernetes.io/projected/aca45b43-9692-46b9-8361-6db806d803e6-kube-api-access-zs4cr\") pod \"must-gather-58jrl\" (UID: \"aca45b43-9692-46b9-8361-6db806d803e6\") " pod="openshift-must-gather-grfnp/must-gather-58jrl"
Jul 28 21:43:08.658446 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.658280    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aca45b43-9692-46b9-8361-6db806d803e6-must-gather-output\") pod \"must-gather-58jrl\" (UID: \"aca45b43-9692-46b9-8361-6db806d803e6\") " pod="openshift-must-gather-grfnp/must-gather-58jrl"
Jul 28 21:43:08.759428 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.759388    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aca45b43-9692-46b9-8361-6db806d803e6-must-gather-output\") pod \"must-gather-58jrl\" (UID: \"aca45b43-9692-46b9-8361-6db806d803e6\") " pod="openshift-must-gather-grfnp/must-gather-58jrl"
Jul 28 21:43:08.759612 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.759556    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs4cr\" (UniqueName: \"kubernetes.io/projected/aca45b43-9692-46b9-8361-6db806d803e6-kube-api-access-zs4cr\") pod \"must-gather-58jrl\" (UID: \"aca45b43-9692-46b9-8361-6db806d803e6\") " pod="openshift-must-gather-grfnp/must-gather-58jrl"
Jul 28 21:43:08.759781 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.759763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aca45b43-9692-46b9-8361-6db806d803e6-must-gather-output\") pod \"must-gather-58jrl\" (UID: \"aca45b43-9692-46b9-8361-6db806d803e6\") " pod="openshift-must-gather-grfnp/must-gather-58jrl"
Jul 28 21:43:08.769075 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.769053    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs4cr\" (UniqueName: \"kubernetes.io/projected/aca45b43-9692-46b9-8361-6db806d803e6-kube-api-access-zs4cr\") pod \"must-gather-58jrl\" (UID: \"aca45b43-9692-46b9-8361-6db806d803e6\") " pod="openshift-must-gather-grfnp/must-gather-58jrl"
Jul 28 21:43:08.945062 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:08.945028    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grfnp/must-gather-58jrl"
Jul 28 21:43:09.082070 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:09.082019    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grfnp/must-gather-58jrl"]
Jul 28 21:43:09.084354 ip-10-0-134-170 kubenswrapper[2565]: W0728 21:43:09.084322    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaca45b43_9692_46b9_8361_6db806d803e6.slice/crio-719d2301d96a1c6eac731b1b5280f02ac210913383c8f4cd695ed5f8a692360e WatchSource:0}: Error finding container 719d2301d96a1c6eac731b1b5280f02ac210913383c8f4cd695ed5f8a692360e: Status 404 returned error can't find the container with id 719d2301d96a1c6eac731b1b5280f02ac210913383c8f4cd695ed5f8a692360e
Jul 28 21:43:09.086256 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:09.086236    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 21:43:09.231352 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:09.231263    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grfnp/must-gather-58jrl" event={"ID":"aca45b43-9692-46b9-8361-6db806d803e6","Type":"ContainerStarted","Data":"719d2301d96a1c6eac731b1b5280f02ac210913383c8f4cd695ed5f8a692360e"}
Jul 28 21:43:10.243998 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:10.243108    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grfnp/must-gather-58jrl" event={"ID":"aca45b43-9692-46b9-8361-6db806d803e6","Type":"ContainerStarted","Data":"4319f0473619fe4fb91d89a8f0819f6abbbd197683337762dda2190e0f3cafeb"}
Jul 28 21:43:10.243998 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:10.243166    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grfnp/must-gather-58jrl" event={"ID":"aca45b43-9692-46b9-8361-6db806d803e6","Type":"ContainerStarted","Data":"b995f4d78efd1b76a4d7cc1077e9bbd1a7d7624c55b0db194da48b74ac29d54b"}
Jul 28 21:43:10.259980 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:10.259929    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-grfnp/must-gather-58jrl" podStartSLOduration=1.4443525400000001 podStartE2EDuration="2.259909838s" totalImagesPullingTime="815.557298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 21:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:43:10.259594943 +0000 UTC m=+3345.483937929" watchObservedRunningTime="2026-07-28 21:43:10.259909838 +0000 UTC m=+3345.484252826"
Jul 28 21:43:11.626889 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:11.626857    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d8rwp_f0b9d1d7-b8e8-4394-8d05-9bbc3ef89f22/global-pull-secret-syncer/0.log"
Jul 28 21:43:11.842901 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:11.842859    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dkngp_50035216-1c15-4307-ba45-212599d1e5ed/konnectivity-agent/0.log"
Jul 28 21:43:11.938449 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:11.938422    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-170.ec2.internal_29284132b693c2f8453092faf219c32b/haproxy/0.log"
Jul 28 21:43:16.319314 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:16.319277    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j_51aa2b30-ff05-45fa-90db-5aa6bf4970ae/extract/0.log"
Jul 28 21:43:16.355754 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:16.355725    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j_51aa2b30-ff05-45fa-90db-5aa6bf4970ae/util/0.log"
Jul 28 21:43:16.388274 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:16.388176    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gfs8j_51aa2b30-ff05-45fa-90db-5aa6bf4970ae/pull/0.log"
Jul 28 21:43:16.418768 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:16.418689    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz_7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4/extract/0.log"
Jul 28 21:43:16.453183 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:16.452987    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz_7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4/util/0.log"
Jul 28 21:43:16.492063 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:16.492021    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0d47sz_7ffaf5af-3b4e-41e8-9e3e-bb3490d559b4/pull/0.log"
Jul 28 21:43:16.526866 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:16.526828    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn_073b3df5-255a-43d8-bdf8-1e75b767507a/extract/0.log"
Jul 28 21:43:16.560723 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:16.560694    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn_073b3df5-255a-43d8-bdf8-1e75b767507a/util/0.log"
Jul 28 21:43:16.589227 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:16.589126    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736gzsn_073b3df5-255a-43d8-bdf8-1e75b767507a/pull/0.log"
Jul 28 21:43:16.625825 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:16.625791    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn_ed38cb3e-13af-4f80-b366-065ffe40d3f4/extract/0.log"
Jul 28 21:43:16.653410 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:16.653379    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn_ed38cb3e-13af-4f80-b366-065ffe40d3f4/util/0.log"
Jul 28 21:43:16.681992 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:16.681963    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19xgwn_ed38cb3e-13af-4f80-b366-065ffe40d3f4/pull/0.log"
Jul 28 21:43:17.149282 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:17.149250    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-6zq5m_f94621ee-e112-4507-b648-81c49a3d370b/kuadrant-console-plugin/0.log"
Jul 28 21:43:17.507692 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:17.507666    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-nbrxn_37a1bb31-c8f7-465e-8610-e7ae408a4a74/limitador/0.log"
Jul 28 21:43:19.559942 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:19.559906    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-rvtwd_18a1dfd7-9d6b-4bb8-a22c-956fca90189d/cluster-monitoring-operator/0.log"
Jul 28 21:43:19.897113 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:19.897024    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t5fvv_3568e7d8-5fff-4276-83d3-b2d9400d4d40/node-exporter/0.log"
Jul 28 21:43:19.923113 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:19.923085    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t5fvv_3568e7d8-5fff-4276-83d3-b2d9400d4d40/kube-rbac-proxy/0.log"
Jul 28 21:43:19.952146 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:19.952096    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t5fvv_3568e7d8-5fff-4276-83d3-b2d9400d4d40/init-textfile/0.log"
Jul 28 21:43:19.982540 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:19.982512    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-42cwb_6525614c-96ff-4222-83eb-a492804d204a/kube-rbac-proxy-main/0.log"
Jul 28 21:43:20.007035 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:20.007004    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-42cwb_6525614c-96ff-4222-83eb-a492804d204a/kube-rbac-proxy-self/0.log"
Jul 28 21:43:20.031961 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:20.031926    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-42cwb_6525614c-96ff-4222-83eb-a492804d204a/openshift-state-metrics/0.log"
Jul 28 21:43:20.401995 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:20.401964    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64d969f794-5f2bn_ec76b3bd-91e4-4dcd-a677-5abf20db6ed3/thanos-query/0.log"
Jul 28 21:43:20.425072 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:20.425043    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64d969f794-5f2bn_ec76b3bd-91e4-4dcd-a677-5abf20db6ed3/kube-rbac-proxy-web/0.log"
Jul 28 21:43:20.453102 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:20.453071    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64d969f794-5f2bn_ec76b3bd-91e4-4dcd-a677-5abf20db6ed3/kube-rbac-proxy/0.log"
Jul 28 21:43:20.477248 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:20.477219    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64d969f794-5f2bn_ec76b3bd-91e4-4dcd-a677-5abf20db6ed3/prom-label-proxy/0.log"
Jul 28 21:43:20.500810 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:20.500781    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64d969f794-5f2bn_ec76b3bd-91e4-4dcd-a677-5abf20db6ed3/kube-rbac-proxy-rules/0.log"
Jul 28 21:43:20.523671 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:20.523625    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-64d969f794-5f2bn_ec76b3bd-91e4-4dcd-a677-5abf20db6ed3/kube-rbac-proxy-metrics/0.log"
Jul 28 21:43:21.636345 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:21.636302    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-wsvnr_4c15ac9e-5a4c-4110-93d8-4217a56ea81b/networking-console-plugin/0.log"
Jul 28 21:43:21.856048 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:21.856016    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"]
Jul 28 21:43:21.864571 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:21.864544    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"
Jul 28 21:43:21.868473 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:21.868436    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"]
Jul 28 21:43:21.992243 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:21.992207    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6ea928b-fd8d-44d7-9c8b-c36b73833c07-sys\") pod \"perf-node-gather-daemonset-8zcgp\" (UID: \"b6ea928b-fd8d-44d7-9c8b-c36b73833c07\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"
Jul 28 21:43:21.992531 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:21.992513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6ea928b-fd8d-44d7-9c8b-c36b73833c07-lib-modules\") pod \"perf-node-gather-daemonset-8zcgp\" (UID: \"b6ea928b-fd8d-44d7-9c8b-c36b73833c07\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"
Jul 28 21:43:21.992780 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:21.992758    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96wcj\" (UniqueName: \"kubernetes.io/projected/b6ea928b-fd8d-44d7-9c8b-c36b73833c07-kube-api-access-96wcj\") pod \"perf-node-gather-daemonset-8zcgp\" (UID: \"b6ea928b-fd8d-44d7-9c8b-c36b73833c07\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"
Jul 28 21:43:21.992966 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:21.992950    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b6ea928b-fd8d-44d7-9c8b-c36b73833c07-proc\") pod \"perf-node-gather-daemonset-8zcgp\" (UID: \"b6ea928b-fd8d-44d7-9c8b-c36b73833c07\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"
Jul 28 21:43:21.993141 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:21.993126    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b6ea928b-fd8d-44d7-9c8b-c36b73833c07-podres\") pod \"perf-node-gather-daemonset-8zcgp\" (UID: \"b6ea928b-fd8d-44d7-9c8b-c36b73833c07\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"
Jul 28 21:43:22.094042 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:22.094001    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96wcj\" (UniqueName: \"kubernetes.io/projected/b6ea928b-fd8d-44d7-9c8b-c36b73833c07-kube-api-access-96wcj\") pod \"perf-node-gather-daemonset-8zcgp\" (UID: \"b6ea928b-fd8d-44d7-9c8b-c36b73833c07\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"
Jul 28 21:43:22.094764 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:22.094737    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b6ea928b-fd8d-44d7-9c8b-c36b73833c07-proc\") pod \"perf-node-gather-daemonset-8zcgp\" (UID: \"b6ea928b-fd8d-44d7-9c8b-c36b73833c07\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"
Jul 28 21:43:22.095068 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:22.095050    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b6ea928b-fd8d-44d7-9c8b-c36b73833c07-podres\") pod \"perf-node-gather-daemonset-8zcgp\" (UID: \"b6ea928b-fd8d-44d7-9c8b-c36b73833c07\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"
Jul 28 21:43:22.095383 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:22.095366    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6ea928b-fd8d-44d7-9c8b-c36b73833c07-sys\") pod \"perf-node-gather-daemonset-8zcgp\" (UID: \"b6ea928b-fd8d-44d7-9c8b-c36b73833c07\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"
Jul 28 21:43:22.095589 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:22.095573    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6ea928b-fd8d-44d7-9c8b-c36b73833c07-lib-modules\") pod \"perf-node-gather-daemonset-8zcgp\" (UID: \"b6ea928b-fd8d-44d7-9c8b-c36b73833c07\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"
Jul 28 21:43:22.095789 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:22.095759    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b6ea928b-fd8d-44d7-9c8b-c36b73833c07-lib-modules\") pod \"perf-node-gather-daemonset-8zcgp\" (UID: \"b6ea928b-fd8d-44d7-9c8b-c36b73833c07\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"
Jul 28 21:43:22.095905 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:22.095530    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b6ea928b-fd8d-44d7-9c8b-c36b73833c07-sys\") pod \"perf-node-gather-daemonset-8zcgp\" (UID: \"b6ea928b-fd8d-44d7-9c8b-c36b73833c07\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"
Jul 28 21:43:22.095905 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:22.095287    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b6ea928b-fd8d-44d7-9c8b-c36b73833c07-podres\") pod \"perf-node-gather-daemonset-8zcgp\" (UID: \"b6ea928b-fd8d-44d7-9c8b-c36b73833c07\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"
Jul 28 21:43:22.095905 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:22.094984    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b6ea928b-fd8d-44d7-9c8b-c36b73833c07-proc\") pod \"perf-node-gather-daemonset-8zcgp\" (UID: \"b6ea928b-fd8d-44d7-9c8b-c36b73833c07\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"
Jul 28 21:43:22.103339 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:22.103306    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96wcj\" (UniqueName: \"kubernetes.io/projected/b6ea928b-fd8d-44d7-9c8b-c36b73833c07-kube-api-access-96wcj\") pod \"perf-node-gather-daemonset-8zcgp\" (UID: \"b6ea928b-fd8d-44d7-9c8b-c36b73833c07\") " pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"
Jul 28 21:43:22.177701 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:22.177651    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"
Jul 28 21:43:22.363743 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:22.362298    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"]
Jul 28 21:43:22.831140 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:22.831112    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8666db5475-r86qc_769c953d-69e2-4ccd-9017-ec62469eb8aa/console/0.log"
Jul 28 21:43:22.872289 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:22.872256    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-6bkcp_eb524232-3306-4310-968b-178c376138e8/download-server/0.log"
Jul 28 21:43:23.363926 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:23.363888    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp" event={"ID":"b6ea928b-fd8d-44d7-9c8b-c36b73833c07","Type":"ContainerStarted","Data":"65b26494a4c2a30ad6bca1cf5d49ba7e719fc6375593e3e470929be117a6fc46"}
Jul 28 21:43:23.364111 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:23.363948    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp" event={"ID":"b6ea928b-fd8d-44d7-9c8b-c36b73833c07","Type":"ContainerStarted","Data":"0d222411524e0a0f00460320a68644e09b9868b60387f653fc80ae21c54851d9"}
Jul 28 21:43:23.364111 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:23.364075    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"
Jul 28 21:43:23.383628 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:23.383562    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp" podStartSLOduration=2.383541489 podStartE2EDuration="2.383541489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 21:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 21:43:23.382144907 +0000 UTC m=+3358.606487895" watchObservedRunningTime="2026-07-28 21:43:23.383541489 +0000 UTC m=+3358.607884476"
Jul 28 21:43:24.588369 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:24.588339    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x6qn9_41a72d4d-a128-4ed4-934f-0aebbe9825b6/dns/0.log"
Jul 28 21:43:24.613241 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:24.613217    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x6qn9_41a72d4d-a128-4ed4-934f-0aebbe9825b6/kube-rbac-proxy/0.log"
Jul 28 21:43:24.664537 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:24.664510    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7prgk_ff2108d7-6551-4eaa-8c51-a234010bc4ba/dns-node-resolver/0.log"
Jul 28 21:43:25.254703 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:25.254672    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-586cdbddd-fbphg_1ff0086b-af29-49c2-9569-ee3b382929b3/registry/0.log"
Jul 28 21:43:25.281766 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:25.281732    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5ngbd_c20cff94-12ac-4cfe-861f-cb7557a7a0e6/node-ca/0.log"
Jul 28 21:43:26.560733 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:26.560705    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68cb4f9b6-gxtld_19df36da-d263-4e8f-8589-45e88f835635/kube-auth-proxy/0.log"
Jul 28 21:43:27.572676 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:27.572628    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7g9v6_4692cbb4-e2af-4b0d-ab5e-ff1be75e2996/serve-healthcheck-canary/0.log"
Jul 28 21:43:27.585524 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:27.585456    2565 scope.go:117] "RemoveContainer" containerID="360f39b4de2f7f7e8e1ca994be34c5aa5604f94d3f75b78a3b3cc1119a93e098"
Jul 28 21:43:27.595571 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:27.595547    2565 scope.go:117] "RemoveContainer" containerID="0fc0e40278628b801d63d3cd2aaa2a3ba5119a928dc99d94ed76e75c27e04385"
Jul 28 21:43:28.141054 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:28.141021    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7vtw8_100dd861-fcc1-45da-ac5b-0797754c195e/kube-rbac-proxy/0.log"
Jul 28 21:43:28.172575 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:28.172534    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7vtw8_100dd861-fcc1-45da-ac5b-0797754c195e/exporter/0.log"
Jul 28 21:43:28.197001 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:28.196972    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7vtw8_100dd861-fcc1-45da-ac5b-0797754c195e/extractor/0.log"
Jul 28 21:43:29.382892 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:29.382864    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-grfnp/perf-node-gather-daemonset-8zcgp"
Jul 28 21:43:31.057180 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:31.057141    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mdl2q_c1831985-93ba-4383-bcb3-08079db37381/manager/0.log"
Jul 28 21:43:31.086705 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:31.086608    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cc774576c-57mmd_b2545b75-760e-498e-b20c-f417090b67d0/manager/0.log"
Jul 28 21:43:31.112955 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:31.112926    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cc774576c-57mmd_b2545b75-760e-498e-b20c-f417090b67d0/copy-manifests/0.log"
Jul 28 21:43:32.492427 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:32.492400    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68cb4f9b6-gxtld_19df36da-d263-4e8f-8589-45e88f835635/kube-auth-proxy/0.log"
Jul 28 21:43:32.615426 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:32.615389    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-bb9684f66-86jlg_eff5404c-a39a-4980-bf3b-88ea6665c1d8/manager/0.log"
Jul 28 21:43:32.674956 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:32.674926    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-gldg9_02c59153-c394-4b59-adbc-1e6416b3cea2/openshift-lws-operator/0.log"
Jul 28 21:43:34.342374 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:34.342344    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68cb4f9b6-gxtld_19df36da-d263-4e8f-8589-45e88f835635/kube-auth-proxy/0.log"
Jul 28 21:43:37.831051 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:37.831021    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-cgkpx_4c5e2247-0906-44a8-b074-f9f03891fff5/migrator/0.log"
Jul 28 21:43:37.855212 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:37.855185    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-cgkpx_4c5e2247-0906-44a8-b074-f9f03891fff5/graceful-termination/0.log"
Jul 28 21:43:39.466290 ip-10-0-134-170 kubenswrapper[2565]: I0728 21:43:39.466258    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k7vsq_b81437ec-1218-4d56-9dc0-5f394a5e45f4/kube-multus-additional-cni-plugins/0.log"