Jul 28 11:28:47.463905 ip-10-0-141-155 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 11:28:47.463914 ip-10-0-141-155 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 11:28:47.463921 ip-10-0-141-155 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 11:28:47.464337 ip-10-0-141-155 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 11:28:57.713148 ip-10-0-141-155 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 11:28:57.713168 ip-10-0-141-155 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 399d3abc04aa439591bdf3d06d880516 --
Jul 28 11:30:46.637427 ip-10-0-141-155 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 11:30:46.953173 ip-10-0-141-155 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 11:30:46.953173 ip-10-0-141-155 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 11:30:46.953173 ip-10-0-141-155 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 11:30:46.953173 ip-10-0-141-155 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 11:30:46.953173 ip-10-0-141-155 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 11:30:46.954145 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.954075    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 11:30:46.959876 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959856    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 11:30:46.959876 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959871    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 11:30:46.959876 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959875    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 11:30:46.959876 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959878    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 11:30:46.959876 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959881    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 11:30:46.959876 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959884    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 11:30:46.960081 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959887    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 11:30:46.960081 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959890    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 11:30:46.960081 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959894    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 11:30:46.960081 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959898    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 11:30:46.960081 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959901    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 11:30:46.960081 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959904    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 11:30:46.960081 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959907    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 11:30:46.960081 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959912    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 11:30:46.960081 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959916    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 11:30:46.960081 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959919    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 11:30:46.960081 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959922    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 11:30:46.960081 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959925    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 11:30:46.960081 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959928    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 11:30:46.960081 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959932    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 11:30:46.960081 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959934    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 11:30:46.960081 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959937    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 11:30:46.960081 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959940    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 11:30:46.960081 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959942    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 11:30:46.960081 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959945    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 11:30:46.960557 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959948    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 11:30:46.960557 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959950    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 11:30:46.960557 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959953    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 11:30:46.960557 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959955    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 11:30:46.960557 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959958    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 11:30:46.960557 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959961    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 11:30:46.960557 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959963    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 11:30:46.960557 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959966    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 11:30:46.960557 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959969    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 11:30:46.960557 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959971    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 11:30:46.960557 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959974    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 11:30:46.960557 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959977    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 11:30:46.960557 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959979    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 11:30:46.960557 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959982    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 11:30:46.960557 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959986    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 11:30:46.960557 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959988    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 11:30:46.960557 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959992    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 11:30:46.960557 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959995    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 11:30:46.960557 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.959998    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 11:30:46.961011 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960000    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 11:30:46.961011 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960003    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 11:30:46.961011 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960006    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 11:30:46.961011 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960008    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 11:30:46.961011 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960011    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 11:30:46.961011 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960013    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 11:30:46.961011 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960016    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 11:30:46.961011 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960019    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 11:30:46.961011 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960022    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 11:30:46.961011 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960024    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 11:30:46.961011 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960027    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 11:30:46.961011 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960029    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 11:30:46.961011 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960032    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 11:30:46.961011 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960034    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 11:30:46.961011 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960037    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 11:30:46.961011 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960039    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 11:30:46.961011 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960042    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 11:30:46.961011 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960044    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 11:30:46.961011 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960047    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 11:30:46.961011 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960050    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 11:30:46.961520 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960052    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 11:30:46.961520 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960055    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 11:30:46.961520 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960058    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 11:30:46.961520 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960060    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 11:30:46.961520 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960063    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 11:30:46.961520 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960066    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 11:30:46.961520 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960068    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 11:30:46.961520 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960072    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 11:30:46.961520 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960075    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 11:30:46.961520 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960077    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 11:30:46.961520 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960080    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 11:30:46.961520 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960083    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 11:30:46.961520 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960086    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 11:30:46.961520 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960089    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 11:30:46.961520 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960092    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 11:30:46.961520 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960094    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 11:30:46.961520 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960097    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 11:30:46.961520 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960099    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 11:30:46.961520 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960102    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 11:30:46.961520 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960107    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 11:30:46.962007 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960109    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 11:30:46.962007 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960112    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 11:30:46.962007 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960115    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 11:30:46.962007 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960118    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 11:30:46.962007 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960122    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 11:30:46.962007 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960125    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 11:30:46.962007 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960128    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 11:30:46.962007 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960130    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 11:30:46.962007 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960133    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 11:30:46.962007 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960135    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 11:30:46.962007 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960138    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 11:30:46.962007 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960140    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 11:30:46.962007 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960143    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 11:30:46.962007 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960145    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 11:30:46.962007 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960148    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 11:30:46.962007 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960832    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 11:30:46.962007 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960838    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 11:30:46.962007 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960841    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 11:30:46.962007 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960844    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 11:30:46.962007 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960847    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 11:30:46.962511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960852    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 11:30:46.962511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960855    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 11:30:46.962511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960858    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 11:30:46.962511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960861    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 11:30:46.962511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960864    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 11:30:46.962511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960866    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 11:30:46.962511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960869    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 11:30:46.962511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960871    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 11:30:46.962511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960874    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 11:30:46.962511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960876    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 11:30:46.962511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960879    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 11:30:46.962511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960882    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 11:30:46.962511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960885    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 11:30:46.962511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960888    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 11:30:46.962511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960890    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 11:30:46.962511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960892    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 11:30:46.962511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960895    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 11:30:46.962511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960897    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 11:30:46.962511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960901    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 11:30:46.962511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960903    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 11:30:46.963102 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960906    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 11:30:46.963102 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960908    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 11:30:46.963102 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960910    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 11:30:46.963102 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960914    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 11:30:46.963102 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960917    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 11:30:46.963102 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960919    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 11:30:46.963102 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960922    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 11:30:46.963102 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960925    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 11:30:46.963102 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960927    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 11:30:46.963102 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960931    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 11:30:46.963102 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960934    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 11:30:46.963102 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960936    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 11:30:46.963102 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960939    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 11:30:46.963102 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960941    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 11:30:46.963102 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960944    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 11:30:46.963102 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960947    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 11:30:46.963102 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960949    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 11:30:46.963102 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960952    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 11:30:46.963102 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960954    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 11:30:46.963102 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960957    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 11:30:46.963617 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960959    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 11:30:46.963617 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960962    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 11:30:46.963617 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960964    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 11:30:46.963617 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960967    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 11:30:46.963617 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960969    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 11:30:46.963617 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960972    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 11:30:46.963617 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960975    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 11:30:46.963617 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960977    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 11:30:46.963617 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960979    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 11:30:46.963617 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960982    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 11:30:46.963617 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960985    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 11:30:46.963617 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960989    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 11:30:46.963617 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960992    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 11:30:46.963617 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960995    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 11:30:46.963617 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.960998    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 11:30:46.963617 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961001    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 11:30:46.963617 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961004    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 11:30:46.963617 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961007    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 11:30:46.963617 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961010    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 11:30:46.963617 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961013    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 11:30:46.964092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961015    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 11:30:46.964092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961019    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 11:30:46.964092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961021    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 11:30:46.964092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961025    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 11:30:46.964092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961027    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 11:30:46.964092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961031    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 11:30:46.964092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961034    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 11:30:46.964092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961036    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 11:30:46.964092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961039    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 11:30:46.964092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961041    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 11:30:46.964092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961044    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 11:30:46.964092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961047    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 11:30:46.964092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961050    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 11:30:46.964092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961053    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 11:30:46.964092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961055    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 11:30:46.964092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961058    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 11:30:46.964092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961061    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 11:30:46.964092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961063    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 11:30:46.964092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961066    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 11:30:46.964092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961069    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 11:30:46.964627 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961071    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 11:30:46.964627 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961074    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 11:30:46.964627 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961076    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 11:30:46.964627 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961079    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 11:30:46.964627 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961081    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 11:30:46.964627 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961084    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 11:30:46.964627 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961088    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 11:30:46.964627 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961091    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 11:30:46.964627 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961094    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 11:30:46.964627 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961097    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 11:30:46.964627 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961100    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 11:30:46.964627 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961102    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 11:30:46.964627 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961105    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 11:30:46.964627 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961108    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 11:30:46.964627 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961473    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 11:30:46.964627 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961481    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 11:30:46.964627 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961483    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 11:30:46.964627 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961486    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 11:30:46.964627 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961489    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 11:30:46.965092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961494    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 11:30:46.965092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961498    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 11:30:46.965092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961501    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 11:30:46.965092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961505    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 11:30:46.965092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961508    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 11:30:46.965092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961511    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 11:30:46.965092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961514    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 11:30:46.965092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961517    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 11:30:46.965092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961521    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 11:30:46.965092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961524    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 11:30:46.965092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961527    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 11:30:46.965092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961531    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 11:30:46.965092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961534    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 11:30:46.965092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961537    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 11:30:46.965092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961540    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 11:30:46.965092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961543    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 11:30:46.965092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961546    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 11:30:46.965092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961548    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 11:30:46.965092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961551    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 11:30:46.965566 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961554    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 11:30:46.965566 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961556    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 11:30:46.965566 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961559    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 11:30:46.965566 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961561    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 11:30:46.965566 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961565    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 11:30:46.965566 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961567    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 11:30:46.965566 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961570    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 11:30:46.965566 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961572    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 11:30:46.965566 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961575    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 11:30:46.965566 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961578    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 11:30:46.965566 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961581    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 11:30:46.965566 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961584    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 11:30:46.965566 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961586    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 11:30:46.965566 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961589    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 11:30:46.965566 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961591    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 11:30:46.965566 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961594    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 11:30:46.965566 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961597    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 11:30:46.965566 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961599    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 11:30:46.965566 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961604    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 11:30:46.966060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961607    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 11:30:46.966060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961609    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 11:30:46.966060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961612    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 11:30:46.966060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961615    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 11:30:46.966060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961617    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 11:30:46.966060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961620    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 11:30:46.966060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961622    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 11:30:46.966060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961625    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 11:30:46.966060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961627    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 11:30:46.966060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961630    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 11:30:46.966060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961633    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 11:30:46.966060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961635    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 11:30:46.966060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961638    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 11:30:46.966060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961640    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 11:30:46.966060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961643    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 11:30:46.966060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961645    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 11:30:46.966060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961648    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 11:30:46.966060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961650    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 11:30:46.966060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961653    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 11:30:46.966060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961655    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 11:30:46.966560 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961657    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 11:30:46.966560 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961660    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 11:30:46.966560 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961664    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 11:30:46.966560 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961666    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 11:30:46.966560 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961669    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 11:30:46.966560 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961671    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 11:30:46.966560 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961674    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 11:30:46.966560 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961676    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 11:30:46.966560 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961679    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 11:30:46.966560 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961682    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 11:30:46.966560 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961684    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 11:30:46.966560 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961688    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 11:30:46.966560 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961690    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 11:30:46.966560 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961693    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 11:30:46.966560 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961696    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 11:30:46.966560 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961699    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 11:30:46.966560 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961701    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 11:30:46.966560 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961705    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 11:30:46.966560 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961717    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 11:30:46.966560 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961721    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961724    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961727    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961730    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961733    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961735    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961738    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961741    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961744    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961746    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961749    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961752    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961754    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961757    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961759    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961763    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.961766    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961825    2561 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961832    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961838    2561 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961842    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 11:30:46.967080 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961849    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961852    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961856    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961862    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961865    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961868    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961871    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961875    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961878    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961881    2561 flags.go:64] FLAG: --cgroup-root=""
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961884    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961887    2561 flags.go:64] FLAG: --client-ca-file=""
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961890    2561 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961893    2561 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961896    2561 flags.go:64] FLAG: --cluster-domain=""
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961899    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961903    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961906    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961909    2561 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961913    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961916    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961919    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961922    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961925    2561 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 11:30:46.967621 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961928    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961931    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961935    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961938    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961942    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961945    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961948    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961951    2561 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961954    2561 flags.go:64] FLAG: --enable-server="true"
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961956    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961960    2561 flags.go:64] FLAG: --event-burst="100"
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961965    2561 flags.go:64] FLAG: --event-qps="50"
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961968    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961971    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961974    2561 flags.go:64] FLAG: --eviction-hard=""
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961977    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961980    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961983    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961986    2561 flags.go:64] FLAG: --eviction-soft=""
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961988    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961991    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961994    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.961997    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962000    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962003    2561 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 11:30:46.968219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962006    2561 flags.go:64] FLAG: --feature-gates=""
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962009    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962013    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962016    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962019    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962022    2561 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962025    2561 flags.go:64] FLAG: --help="false"
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962027    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-141-155.ec2.internal"
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962030    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962034    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962037    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962040    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962044    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962046    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962049    2561 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962052    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962055    2561 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962057    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962062    2561 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962065    2561 flags.go:64] FLAG: --kube-reserved=""
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962068    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962070    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962073    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962076    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 11:30:46.968877 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962079    2561 flags.go:64] FLAG: --lock-file=""
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962082    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962085    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962087    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962093    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962096    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962098    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962101    2561 flags.go:64] FLAG: --logging-format="text"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962104    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962107    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962110    2561 flags.go:64] FLAG: --manifest-url=""
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962112    2561 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962117    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962119    2561 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962123    2561 flags.go:64] FLAG: --max-pods="110"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962126    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962129    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962132    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962135    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962137    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962141    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962143    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962150    2561 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962165    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962169    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 11:30:46.969485 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962172    2561 flags.go:64] FLAG: --pod-cidr=""
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962176    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962182    2561 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962184    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962187    2561 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962190    2561 flags.go:64] FLAG: --port="10250"
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962194    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962196    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0619b5a5e944a6039"
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962200    2561 flags.go:64] FLAG: --qos-reserved=""
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962203    2561 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962206    2561 flags.go:64] FLAG: --register-node="true"
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962208    2561 flags.go:64] FLAG: --register-with-taints=""
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962212    2561 flags.go:64] FLAG: --registry-burst="10"
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962216    2561 flags.go:64] FLAG: --registry-qps="5"
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962218    2561 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962221    2561 flags.go:64] FLAG: --reserved-memory=""
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962224    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962228    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962230    2561 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962233    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962236    2561 flags.go:64] FLAG: --runonce="false"
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962239    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962242    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962245    2561 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962248    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 11:30:46.970106 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962251    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962254    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962256    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962259    2561 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962262    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962265    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962268    2561 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962271    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962274    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962277    2561 flags.go:64] FLAG: --system-cgroups=""
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962280    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962285    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962288    2561 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962291    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962294    2561 flags.go:64] FLAG: --tls-min-version=""
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962297    2561 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962300    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962302    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962305    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962308    2561 flags.go:64] FLAG: --v="2"
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962312    2561 flags.go:64] FLAG: --version="false"
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962316    2561 flags.go:64] FLAG: --vmodule=""
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962320    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.962323    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962429    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 11:30:46.970751 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962433    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 11:30:46.971375 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962436    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 11:30:46.971375 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962439    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 11:30:46.971375 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962441    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 11:30:46.971375 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962444    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 11:30:46.971375 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962447    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 11:30:46.971375 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962449    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 11:30:46.971375 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962452    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 11:30:46.971375 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962455    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 11:30:46.971375 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962457    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 11:30:46.971375 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962460    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 11:30:46.971375 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962462    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 11:30:46.971375 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962464    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 11:30:46.971375 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962467    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 11:30:46.971375 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962469    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 11:30:46.971375 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962473    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 11:30:46.971375 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962476    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 11:30:46.971375 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962478    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 11:30:46.971375 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962481    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 11:30:46.971375 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962483    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 11:30:46.971375 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962486    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 11:30:46.971869 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962488    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 11:30:46.971869 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962491    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 11:30:46.971869 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962493    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 11:30:46.971869 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962496    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 11:30:46.971869 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962498    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 11:30:46.971869 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962501    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 11:30:46.971869 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962503    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 11:30:46.971869 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962505    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 11:30:46.971869 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962512    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 11:30:46.971869 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962515    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 11:30:46.971869 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962518    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 11:30:46.971869 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962521    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 11:30:46.971869 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962523    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 11:30:46.971869 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962526    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 11:30:46.971869 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962529    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 11:30:46.971869 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962531    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 11:30:46.971869 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962534    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 11:30:46.971869 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962536    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 11:30:46.971869 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962541    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 11:30:46.971869 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962543    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 11:30:46.972366 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962546    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 11:30:46.972366 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962549    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 11:30:46.972366 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962552    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 11:30:46.972366 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962554    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 11:30:46.972366 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962557    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 11:30:46.972366 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962559    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 11:30:46.972366 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962563    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 11:30:46.972366 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962565    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 11:30:46.972366 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962568    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 11:30:46.972366 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962571    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 11:30:46.972366 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962575    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 11:30:46.972366 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962578    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 11:30:46.972366 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962581    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 11:30:46.972366 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962584    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 11:30:46.972366 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962587    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 11:30:46.972366 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962590    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 11:30:46.972366 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962593    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 11:30:46.972366 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962595    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 11:30:46.972366 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962598    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 11:30:46.972832 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962601    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 11:30:46.972832 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962603    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 11:30:46.972832 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962611    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 11:30:46.972832 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962614    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 11:30:46.972832 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962616    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 11:30:46.972832 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962619    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 11:30:46.972832 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962621    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 11:30:46.972832 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962624    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 11:30:46.972832 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962626    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 11:30:46.972832 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962629    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 11:30:46.972832 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962631    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 11:30:46.972832 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962635    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 11:30:46.972832 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962638    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 11:30:46.972832 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962641    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 11:30:46.972832 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962645    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 11:30:46.972832 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962648    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 11:30:46.972832 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962651    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 11:30:46.972832 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962653    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 11:30:46.972832 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962656    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 11:30:46.973337 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962659    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 11:30:46.973337 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962662    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 11:30:46.973337 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962665    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 11:30:46.973337 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962667    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 11:30:46.973337 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962670    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 11:30:46.973337 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962673    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 11:30:46.973337 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962676    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 11:30:46.973337 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962678    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 11:30:46.973337 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962681    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 11:30:46.973337 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962683    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 11:30:46.973337 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962686    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 11:30:46.973337 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962688    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 11:30:46.973337 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962692    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 11:30:46.973337 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962695    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 11:30:46.973337 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962698    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 11:30:46.973337 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962701    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 11:30:46.973337 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962704    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 11:30:46.973337 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962706    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 11:30:46.973337 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.962709    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 11:30:46.973843 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.963370    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 11:30:46.973843 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.969737    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 11:30:46.973843 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.969751    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 11:30:46.973843 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.970346    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 11:30:46.973843 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.970354    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 11:30:46.973843 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970421    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 11:30:46.973843 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970425    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 11:30:46.973843 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970428    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 11:30:46.973843 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970432    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 11:30:46.973843 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970434    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 11:30:46.973843 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970437    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 11:30:46.973843 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970440    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 11:30:46.973843 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970442    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 11:30:46.973843 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970445    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 11:30:46.974231 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970448    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 11:30:46.974231 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970451    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 11:30:46.974231 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970453    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 11:30:46.974231 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970456    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 11:30:46.974231 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970458    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 11:30:46.974231 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970461    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 11:30:46.974231 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970463    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 11:30:46.974231 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970466    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 11:30:46.974231 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970469    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 11:30:46.974231 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970471    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 11:30:46.974231 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970474    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 11:30:46.974231 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970476    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 11:30:46.974231 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970479    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 11:30:46.974231 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970481    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 11:30:46.974231 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970484    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 11:30:46.974231 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970487    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 11:30:46.974231 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970489    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 11:30:46.974231 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970492    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 11:30:46.974231 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970494    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 11:30:46.974231 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970497    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 11:30:46.974786 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970501    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 11:30:46.974786 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970504    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 11:30:46.974786 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970508    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 11:30:46.974786 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970511    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 11:30:46.974786 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970514    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 11:30:46.974786 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970517    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 11:30:46.974786 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970520    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 11:30:46.974786 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970523    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 11:30:46.974786 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970526    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 11:30:46.974786 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970529    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 11:30:46.974786 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970532    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 11:30:46.974786 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970534    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 11:30:46.974786 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970537    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 11:30:46.974786 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970539    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 11:30:46.974786 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970542    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 11:30:46.974786 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970544    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 11:30:46.974786 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970547    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 11:30:46.974786 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970550    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 11:30:46.974786 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970552    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 11:30:46.975397 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970555    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 11:30:46.975397 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970558    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 11:30:46.975397 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970560    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 11:30:46.975397 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970563    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 11:30:46.975397 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970566    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 11:30:46.975397 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970568    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 11:30:46.975397 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970571    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 11:30:46.975397 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970574    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 11:30:46.975397 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970576    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 11:30:46.975397 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970579    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 11:30:46.975397 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970581    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 11:30:46.975397 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970584    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 11:30:46.975397 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970587    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 11:30:46.975397 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970590    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 11:30:46.975397 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970595    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 11:30:46.975397 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970598    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 11:30:46.975397 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970602    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 11:30:46.975397 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970605    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 11:30:46.975397 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970608    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 11:30:46.975397 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970611    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 11:30:46.975883 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970614    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 11:30:46.975883 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970617    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 11:30:46.975883 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970620    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 11:30:46.975883 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970623    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 11:30:46.975883 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970626    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 11:30:46.975883 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970629    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 11:30:46.975883 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970632    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 11:30:46.975883 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970634    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 11:30:46.975883 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970637    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 11:30:46.975883 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970639    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 11:30:46.975883 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970642    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 11:30:46.975883 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970644    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 11:30:46.975883 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970647    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 11:30:46.975883 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970649    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 11:30:46.975883 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970652    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 11:30:46.975883 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970654    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 11:30:46.975883 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970657    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 11:30:46.975883 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970660    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 11:30:46.975883 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970664    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 11:30:46.976573 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970667    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 11:30:46.976573 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970669    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 11:30:46.976573 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970672    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 11:30:46.976573 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970674    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 11:30:46.976573 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970677    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 11:30:46.976573 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970680    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 11:30:46.976573 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970683    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 11:30:46.976573 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970686    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 11:30:46.976573 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970689    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 11:30:46.976573 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970691    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 11:30:46.976573 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970694    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 11:30:46.976573 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970696    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 11:30:46.976573 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.970701    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 11:30:46.976573 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970821    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 11:30:46.976573 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970826    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 11:30:46.977221 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970829    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 11:30:46.977221 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970832    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 11:30:46.977221 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970835    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 11:30:46.977221 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970838    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 11:30:46.977221 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970840    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 11:30:46.977221 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970843    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 11:30:46.977221 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970845    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 11:30:46.977221 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970848    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 11:30:46.977221 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970852    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 11:30:46.977221 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970855    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 11:30:46.977221 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970859    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 11:30:46.977221 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970863    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 11:30:46.977221 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970865    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 11:30:46.977221 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970868    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 11:30:46.977221 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970872    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 11:30:46.977221 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970875    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 11:30:46.977221 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970878    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 11:30:46.977221 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970880    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 11:30:46.977221 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970884    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 11:30:46.977823 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970886    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 11:30:46.977823 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970889    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 11:30:46.977823 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970891    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 11:30:46.977823 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970894    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 11:30:46.977823 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970897    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 11:30:46.977823 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970900    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 11:30:46.977823 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970902    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 11:30:46.977823 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970906    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 11:30:46.977823 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970909    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 11:30:46.977823 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970911    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 11:30:46.977823 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970913    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 11:30:46.977823 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970916    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 11:30:46.977823 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970919    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 11:30:46.977823 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970921    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 11:30:46.977823 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970924    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 11:30:46.977823 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970926    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 11:30:46.977823 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970928    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 11:30:46.977823 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970931    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 11:30:46.977823 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970934    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 11:30:46.977823 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970936    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 11:30:46.978464 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970939    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 11:30:46.978464 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970941    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 11:30:46.978464 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970944    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 11:30:46.978464 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970946    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 11:30:46.978464 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970949    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 11:30:46.978464 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970951    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 11:30:46.978464 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970954    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 11:30:46.978464 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970956    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 11:30:46.978464 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970959    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 11:30:46.978464 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970961    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 11:30:46.978464 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970964    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 11:30:46.978464 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970967    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 11:30:46.978464 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970969    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 11:30:46.978464 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970972    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 11:30:46.978464 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970974    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 11:30:46.978464 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970977    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 11:30:46.978464 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970979    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 11:30:46.978464 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970982    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 11:30:46.978464 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970985    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 11:30:46.979060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970987    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 11:30:46.979060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970990    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 11:30:46.979060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970992    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 11:30:46.979060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970995    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 11:30:46.979060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.970997    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 11:30:46.979060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971000    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 11:30:46.979060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971002    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 11:30:46.979060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971005    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 11:30:46.979060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971007    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 11:30:46.979060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971010    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 11:30:46.979060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971012    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 11:30:46.979060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971015    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 11:30:46.979060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971017    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 11:30:46.979060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971019    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 11:30:46.979060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971022    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 11:30:46.979060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971024    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 11:30:46.979060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971027    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 11:30:46.979060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971029    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 11:30:46.979060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971032    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 11:30:46.979060 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971034    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 11:30:46.979708 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971037    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 11:30:46.979708 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971039    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 11:30:46.979708 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971042    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 11:30:46.979708 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971044    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 11:30:46.979708 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971047    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 11:30:46.979708 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971049    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 11:30:46.979708 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971052    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 11:30:46.979708 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971054    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 11:30:46.979708 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971057    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 11:30:46.979708 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971059    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 11:30:46.979708 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971062    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 11:30:46.979708 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971065    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 11:30:46.979708 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971067    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 11:30:46.979708 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971069    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 11:30:46.979708 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971078    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 11:30:46.979708 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971081    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 11:30:46.979708 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971084    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 11:30:46.979708 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971086    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 11:30:46.979708 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:46.971088    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 11:30:46.980472 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.971093    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 11:30:46.980472 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.971223    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 11:30:46.980472 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.973221    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 11:30:46.980472 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.974369    2561 server.go:1019] "Starting client certificate rotation"
Jul 28 11:30:46.980472 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.974492    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 11:30:46.980472 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.975194    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 11:30:46.980472 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.977179    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 11:30:46.980472 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.978742    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 11:30:46.987574 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.987558    2561 log.go:25] "Validated CRI v1 runtime API"
Jul 28 11:30:46.989390 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.989375    2561 log.go:25] "Validated CRI v1 image API"
Jul 28 11:30:46.990192 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.990174    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 11:30:46.992146 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.992130    2561 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 cb0b6b6c-239b-40cf-8dd7-025e66e5238d:/dev/nvme0n1p4 e17d5277-4e30-4c86-acef-7f95e66d97bc:/dev/nvme0n1p3]
Jul 28 11:30:46.992230 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.992146    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 11:30:46.997589 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.997482    2561 manager.go:216] Machine: {Timestamp:2026-07-28 11:30:46.995618544 +0000 UTC m=+0.261698854 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104436 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec250c0001d17e5309ee28a5e9193550 SystemUUID:ec250c00-01d1-7e53-09ee-28a5e9193550 BootID:399d3abc-04aa-4395-91bd-f3d06d880516 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e2:df:f4:83:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e2:df:f4:83:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:ea:18:14:57:fe Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 11:30:46.997589 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.997587    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 11:30:46.997694 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:46.997657    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 11:30:47.000166 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.000136    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 11:30:47.000304 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.000173    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-155.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 11:30:47.000886 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.000877    2561 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 11:30:47.000921 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.000889    2561 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 11:30:47.000921 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.000896    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 11:30:47.001442 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.001432    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 11:30:47.002701 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.002691    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 11:30:47.006611 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.006600    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 11:30:47.006729 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.006719    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 11:30:47.007293 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.007284    2561 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 11:30:47.007329 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.007302    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 11:30:47.007329 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.007316    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 11:30:47.007329 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.007324    2561 kubelet.go:398] "Adding apiserver pod source"
Jul 28 11:30:47.007428 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.007334    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 11:30:47.007642 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.007629    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 11:30:47.007676 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.007656    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 11:30:47.009347 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.009330    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 11:30:47.010726 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.010708    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 11:30:47.010804 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.010755    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 11:30:47.011368 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.011351    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 11:30:47.011368 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.011374    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 11:30:47.011490 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.011383    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 11:30:47.011490 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.011391    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 11:30:47.011490 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.011403    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 11:30:47.011490 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.011411    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 11:30:47.011490 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.011413    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 11:30:47.011490 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.011422    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 11:30:47.011490 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.011441    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 11:30:47.011490 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.011451    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 11:30:47.011490 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.011460    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 11:30:47.011490 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.011471    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 11:30:47.011490 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.011479    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 11:30:47.012070 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.012061    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 11:30:47.012070 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.012070    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 11:30:47.014947 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.014930    2561 server.go:1268] "Started kubelet"
Jul 28 11:30:47.015059 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.015009    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 11:30:47.017814 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.015070    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 11:30:47.017915 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.017898    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 11:30:47.018134 ip-10-0-141-155 systemd[1]: Started Kubernetes Kubelet.
Jul 28 11:30:47.019213 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:47.019071    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-155.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 11:30:47.019213 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:47.019179    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 11:30:47.019464 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.019438    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-155.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 11:30:47.020569 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.020550    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 11:30:47.020964 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.020949    2561 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 11:30:47.022984 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:47.022224    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-155.ec2.internal.18c670ba20c3aed2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-155.ec2.internal,UID:ip-10-0-141-155.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-155.ec2.internal,},FirstTimestamp:2026-07-28 11:30:47.014903506 +0000 UTC m=+0.280983819,LastTimestamp:2026-07-28 11:30:47.014903506 +0000 UTC m=+0.280983819,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-155.ec2.internal,}"
Jul 28 11:30:47.023285 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:47.023268    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 11:30:47.027469 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.027386    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 11:30:47.027540 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:47.027509    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-155.ec2.internal\" not found"
Jul 28 11:30:47.028051 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.028037    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 11:30:47.028150 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.028128    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 11:30:47.028227 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.028046    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 11:30:47.028227 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.028179    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 11:30:47.028332 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.028257    2561 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 11:30:47.028332 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.028266    2561 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 11:30:47.028593 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.028574    2561 factory.go:55] Registering systemd factory
Jul 28 11:30:47.028593 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.028593    2561 factory.go:223] Registration of the systemd container factory successfully
Jul 28 11:30:47.028809 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.028793    2561 factory.go:153] Registering CRI-O factory
Jul 28 11:30:47.028809 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.028808    2561 factory.go:223] Registration of the crio container factory successfully
Jul 28 11:30:47.028957 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.028854    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 11:30:47.028957 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.028896    2561 factory.go:103] Registering Raw factory
Jul 28 11:30:47.028957 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.028908    2561 manager.go:1196] Started watching for new ooms in manager
Jul 28 11:30:47.029829 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.029812    2561 manager.go:349] Starting recovery of all containers
Jul 28 11:30:47.039575 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:47.039549    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-155.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 11:30:47.040292 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:47.040269    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 11:30:47.040567 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.040556    2561 manager.go:354] Recovery completed
Jul 28 11:30:47.044865 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.044854    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 11:30:47.046625 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.046599    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 11:30:47.046686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.046636    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 11:30:47.046686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.046646    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientPID"
Jul 28 11:30:47.047571 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.047557    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 11:30:47.047571 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.047570    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 11:30:47.047656 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.047587    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 11:30:47.047656 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.047570    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 11:30:47.048390 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.048361    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 11:30:47.048487 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.048399    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 11:30:47.048487 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.048419    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientPID"
Jul 28 11:30:47.049784 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.049773    2561 policy_none.go:49] "None policy: Start"
Jul 28 11:30:47.049834 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.049789    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 11:30:47.049834 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.049800    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 11:30:47.050409 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:47.050342    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-155.ec2.internal.18c670ba22a7b283  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-155.ec2.internal,UID:ip-10-0-141-155.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-155.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-155.ec2.internal,},FirstTimestamp:2026-07-28 11:30:47.046623875 +0000 UTC m=+0.312704189,LastTimestamp:2026-07-28 11:30:47.046623875 +0000 UTC m=+0.312704189,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-155.ec2.internal,}"
Jul 28 11:30:47.051917 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.051900    2561 policy_none.go:47] "Start"
Jul 28 11:30:47.066089 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.065987    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x7dnp"
Jul 28 11:30:47.066089 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:47.066030    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-155.ec2.internal.18c670ba22a7f628  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-155.ec2.internal,UID:ip-10-0-141-155.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-155.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-155.ec2.internal,},FirstTimestamp:2026-07-28 11:30:47.046641192 +0000 UTC m=+0.312721503,LastTimestamp:2026-07-28 11:30:47.046641192 +0000 UTC m=+0.312721503,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-155.ec2.internal,}"
Jul 28 11:30:47.074081 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:47.074027    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-155.ec2.internal.18c670ba22a8178c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-155.ec2.internal,UID:ip-10-0-141-155.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-155.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-155.ec2.internal,},FirstTimestamp:2026-07-28 11:30:47.04664974 +0000 UTC m=+0.312730051,LastTimestamp:2026-07-28 11:30:47.04664974 +0000 UTC m=+0.312730051,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-155.ec2.internal,}"
Jul 28 11:30:47.077667 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.077652    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x7dnp"
Jul 28 11:30:47.083807 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:47.083752    2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-155.ec2.internal.18c670ba22a7b283\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-155.ec2.internal.18c670ba22a7b283  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-155.ec2.internal,UID:ip-10-0-141-155.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-155.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-155.ec2.internal,},FirstTimestamp:2026-07-28 11:30:47.046623875 +0000 UTC m=+0.312704189,LastTimestamp:2026-07-28 11:30:47.048384647 +0000 UTC m=+0.314464970,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-155.ec2.internal,}"
Jul 28 11:30:47.097927 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.097912    2561 manager.go:337] "Starting Device Plugin manager"
Jul 28 11:30:47.121082 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:47.097952    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 11:30:47.121082 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.097965    2561 server.go:87] "Starting device plugin registration server"
Jul 28 11:30:47.121082 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.098195    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 11:30:47.121082 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.098207    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 11:30:47.121082 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.098293    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 11:30:47.121082 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.098367    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 11:30:47.121082 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.098374    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 11:30:47.121082 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:47.098893    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 11:30:47.121082 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:47.098935    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-155.ec2.internal\" not found"
Jul 28 11:30:47.131580 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.131559    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 11:30:47.132737 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.132716    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 11:30:47.132818 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.132747    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 11:30:47.132818 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.132769    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 11:30:47.132921 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:47.132806    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 11:30:47.135209 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.135191    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 11:30:47.198499 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.198475    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 11:30:47.204670 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.204625    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 11:30:47.204670 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.204652    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 11:30:47.204670 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.204663    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientPID"
Jul 28 11:30:47.204788 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.204682    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-155.ec2.internal"
Jul 28 11:30:47.213144 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.213127    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-155.ec2.internal"
Jul 28 11:30:47.213210 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:47.213146    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-155.ec2.internal\": node \"ip-10-0-141-155.ec2.internal\" not found"
Jul 28 11:30:47.232282 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:47.232264    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-155.ec2.internal\" not found"
Jul 28 11:30:47.233291 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.233276    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal"]
Jul 28 11:30:47.233338 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.233328    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 11:30:47.234076 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.234057    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 11:30:47.234150 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.234084    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 11:30:47.234150 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.234093    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientPID"
Jul 28 11:30:47.236498 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.236486    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 11:30:47.236586 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.236571    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal"
Jul 28 11:30:47.236626 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.236611    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 11:30:47.237127 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.237113    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 11:30:47.237217 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.237129    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 11:30:47.237217 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.237140    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 11:30:47.237217 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.237149    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 11:30:47.237217 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.237181    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientPID"
Jul 28 11:30:47.237362 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.237167    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientPID"
Jul 28 11:30:47.239510 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.239497    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal"
Jul 28 11:30:47.239550 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.239521    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 11:30:47.240281 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.240266    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 11:30:47.240350 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.240294    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 11:30:47.240350 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:47.240308    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientPID"
Jul 28 11:30:47.265625 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:47.265604    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-155.ec2.internal\" not found" node="ip-10-0-141-155.ec2.internal"
Jul 28 11:30:47.269899 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:47.269884    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-155.ec2.internal\" not found" node="ip-10-0-141-155.ec2.internal"
Jul 28 11:30:48.736775 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:48.736755    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/efb3d0a42d97d596842f3e8617f33a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal\" (UID: \"efb3d0a42d97d596842f3e8617f33a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal"
Jul 28 11:30:48.736868 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:48.736784    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efb3d0a42d97d596842f3e8617f33a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal\" (UID: \"efb3d0a42d97d596842f3e8617f33a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal"
Jul 28 11:30:48.736868 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:48.736809    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8a2ad931bbcb3d6172f2d426a64e2848-config\") pod \"kube-apiserver-proxy-ip-10-0-141-155.ec2.internal\" (UID: \"8a2ad931bbcb3d6172f2d426a64e2848\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal"
Jul 28 11:30:48.739795 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:48.739776    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-155.ec2.internal\" not found"
Jul 28 11:30:48.836926 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:48.836901    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8a2ad931bbcb3d6172f2d426a64e2848-config\") pod \"kube-apiserver-proxy-ip-10-0-141-155.ec2.internal\" (UID: \"8a2ad931bbcb3d6172f2d426a64e2848\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal"
Jul 28 11:30:48.836926 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:48.836922    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8a2ad931bbcb3d6172f2d426a64e2848-config\") pod \"kube-apiserver-proxy-ip-10-0-141-155.ec2.internal\" (UID: \"8a2ad931bbcb3d6172f2d426a64e2848\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal"
Jul 28 11:30:48.837114 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:48.836946    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/efb3d0a42d97d596842f3e8617f33a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal\" (UID: \"efb3d0a42d97d596842f3e8617f33a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal"
Jul 28 11:30:48.837114 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:48.836986    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/efb3d0a42d97d596842f3e8617f33a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal\" (UID: \"efb3d0a42d97d596842f3e8617f33a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal"
Jul 28 11:30:48.837114 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:48.837004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efb3d0a42d97d596842f3e8617f33a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal\" (UID: \"efb3d0a42d97d596842f3e8617f33a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal"
Jul 28 11:30:48.837114 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:48.837046    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efb3d0a42d97d596842f3e8617f33a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal\" (UID: \"efb3d0a42d97d596842f3e8617f33a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal"
Jul 28 11:30:48.840640 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:48.840625    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-155.ec2.internal\" not found"
Jul 28 11:30:48.941540 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:48.941484    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-155.ec2.internal\" not found"
Jul 28 11:30:48.975701 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:48.975664    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal"
Jul 28 11:30:48.981440 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:48.981196    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal"
Jul 28 11:30:49.041847 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:49.041815    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-155.ec2.internal\" not found"
Jul 28 11:30:49.142306 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:49.142281    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-155.ec2.internal\" not found"
Jul 28 11:30:49.242909 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:49.242845    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-155.ec2.internal\" not found"
Jul 28 11:30:49.260143 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:49.260125    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 11:30:49.343572 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:49.343530    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-155.ec2.internal\" not found"
Jul 28 11:30:49.381734 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:49.381704    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 11:30:49.382152 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:49.381845    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 11:30:49.382152 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:49.381889    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 11:30:49.444020 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:49.443997    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-155.ec2.internal\" not found"
Jul 28 11:30:49.489723 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:49.489682    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 11:25:48 +0000 UTC" deadline="2028-03-05 12:13:22.473816023 +0000 UTC"
Jul 28 11:30:49.489847 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:49.489728    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14064h42m32.98410397s"
Jul 28 11:30:49.505537 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:49.505486    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 11:30:49.535392 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:49.535363    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal"
Jul 28 11:30:49.545837 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:49.545814    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 11:30:49.546691 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:49.546675    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal"
Jul 28 11:30:49.555508 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:49.555476    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2ad931bbcb3d6172f2d426a64e2848.slice/crio-447d4c8171753d6fc55c1555e260f95269a7a2f1500b4f75aa7c8f18d534d3ca WatchSource:0}: Error finding container 447d4c8171753d6fc55c1555e260f95269a7a2f1500b4f75aa7c8f18d534d3ca: Status 404 returned error can't find the container with id 447d4c8171753d6fc55c1555e260f95269a7a2f1500b4f75aa7c8f18d534d3ca
Jul 28 11:30:49.555723 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:49.555708    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb3d0a42d97d596842f3e8617f33a18.slice/crio-bfaf60ddc60cd4018d687c3aaa5e1ac6d641b7f096df1ca1191950173e0b7dbc WatchSource:0}: Error finding container bfaf60ddc60cd4018d687c3aaa5e1ac6d641b7f096df1ca1191950173e0b7dbc: Status 404 returned error can't find the container with id bfaf60ddc60cd4018d687c3aaa5e1ac6d641b7f096df1ca1191950173e0b7dbc
Jul 28 11:30:49.560261 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:49.559216    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 11:30:49.562087 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:49.562073    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 11:30:50.011650 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.011627    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 11:30:50.378659 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.378557    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 11:30:50.415746 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.415648    2561 apiserver.go:52] "Watching apiserver"
Jul 28 11:30:50.423191 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.423167    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 11:30:50.423649 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.423627    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96","openshift-cluster-node-tuning-operator/tuned-dffdp","openshift-dns/node-resolver-hpn2d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal","openshift-multus/multus-additional-cni-plugins-wprpw","openshift-network-diagnostics/network-check-target-bcd9g","kube-system/global-pull-secret-syncer-fg8rp","openshift-image-registry/node-ca-pthzf","openshift-multus/multus-ht9lx","openshift-multus/network-metrics-daemon-ss7h8","openshift-network-operator/iptables-alerter-kgjx5","openshift-ovn-kubernetes/ovnkube-node-f2c2r","kube-system/konnectivity-agent-xwtf2"]
Jul 28 11:30:50.426255 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.426230    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:30:50.426348 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:50.426316    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcd9g" podUID="35b9e83d-edd8-4afa-9368-a7a490338b85"
Jul 28 11:30:50.428402 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.428379    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:30:50.428523 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:50.428458    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ss7h8" podUID="85d71f38-a39a-4a1c-90dc-f5b941567052"
Jul 28 11:30:50.430664 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.430637    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.432849 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.432831    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hpn2d"
Jul 28 11:30:50.434927 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.434905    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 11:30:50.435019 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.434940    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xwtf2"
Jul 28 11:30:50.435019 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.434906    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 11:30:50.435192 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.435176    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xtd8s\""
Jul 28 11:30:50.436022 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.436000    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 11:30:50.436922 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.436893    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 11:30:50.437219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.437194    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jgqgc\""
Jul 28 11:30:50.437359 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.437340    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 11:30:50.437625 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.437608    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 11:30:50.437881 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.437832    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 11:30:50.438065 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.438049    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jtfnw\""
Jul 28 11:30:50.438291 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.438271    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.440198 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.440180    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 11:30:50.441197 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.440416    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 11:30:50.441197 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.440498    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 11:30:50.441197 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.440694    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 11:30:50.441197 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.440705    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kc8vp\""
Jul 28 11:30:50.441197 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.440829    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pthzf"
Jul 28 11:30:50.441197 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.440940    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 11:30:50.442846 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.442800    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 11:30:50.442948 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.442876    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 11:30:50.442948 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.442915    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.443953 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.443934    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 11:30:50.446339 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.444146    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pr5s8\""
Jul 28 11:30:50.446339 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.445436    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.446339 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.445437    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-os-release\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.446339 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.445803    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-cni-binary-copy\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.446339 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.445834    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmr7x\" (UniqueName: \"kubernetes.io/projected/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-kube-api-access-wmr7x\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.446339 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.445887    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndmqc\" (UniqueName: \"kubernetes.io/projected/85d71f38-a39a-4a1c-90dc-f5b941567052-kube-api-access-ndmqc\") pod \"network-metrics-daemon-ss7h8\" (UID: \"85d71f38-a39a-4a1c-90dc-f5b941567052\") " pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:30:50.446339 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.445926    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-kubernetes\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.446339 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.445980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e552bdf3-9456-4e29-b6c8-aaad902d9d5c-hosts-file\") pod \"node-resolver-hpn2d\" (UID: \"e552bdf3-9456-4e29-b6c8-aaad902d9d5c\") " pod="openshift-dns/node-resolver-hpn2d"
Jul 28 11:30:50.446339 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.446015    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-cnibin\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.446339 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.446044    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-host\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.446339 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.446112    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b0fd488-b3f4-4e45-8832-a74efd910839-konnectivity-ca\") pod \"konnectivity-agent-xwtf2\" (UID: \"0b0fd488-b3f4-4e45-8832-a74efd910839\") " pod="kube-system/konnectivity-agent-xwtf2"
Jul 28 11:30:50.446339 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.446161    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-var-lib-kubelet\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.446339 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.446191    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-sysctl-conf\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.446972 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.446244    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-run\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.446972 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.446745    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-tuned\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.446972 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.446774    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.446972 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.446818    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-sysconfig\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.446972 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.446852    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-sysctl-d\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.446972 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.446898    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-lib-modules\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.446972 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.446926    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b0fd488-b3f4-4e45-8832-a74efd910839-agent-certs\") pod \"konnectivity-agent-xwtf2\" (UID: \"0b0fd488-b3f4-4e45-8832-a74efd910839\") " pod="kube-system/konnectivity-agent-xwtf2"
Jul 28 11:30:50.446972 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.446955    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d88qq\" (UniqueName: \"kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq\") pod \"network-check-target-bcd9g\" (UID: \"35b9e83d-edd8-4afa-9368-a7a490338b85\") " pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:30:50.447384 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.447005    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-systemd\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.447384 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.447045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-sys\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.447384 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.447131    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.447384 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.447161    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6tms9\""
Jul 28 11:30:50.447384 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.447171    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.447384 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.447201    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs\") pod \"network-metrics-daemon-ss7h8\" (UID: \"85d71f38-a39a-4a1c-90dc-f5b941567052\") " pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:30:50.447384 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.447220    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 11:30:50.447384 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.447260    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-modprobe-d\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.447384 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.447291    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a930d1a2-a540-40bf-bcb9-f4e24dd760af-tmp\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.447384 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.447331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e552bdf3-9456-4e29-b6c8-aaad902d9d5c-tmp-dir\") pod \"node-resolver-hpn2d\" (UID: \"e552bdf3-9456-4e29-b6c8-aaad902d9d5c\") " pod="openshift-dns/node-resolver-hpn2d"
Jul 28 11:30:50.447384 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.447378    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-system-cni-dir\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.447901 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.447404    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6xbs\" (UniqueName: \"kubernetes.io/projected/a930d1a2-a540-40bf-bcb9-f4e24dd760af-kube-api-access-q6xbs\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.447901 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.447422    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 11:30:50.447901 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.447435    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67mhn\" (UniqueName: \"kubernetes.io/projected/e552bdf3-9456-4e29-b6c8-aaad902d9d5c-kube-api-access-67mhn\") pod \"node-resolver-hpn2d\" (UID: \"e552bdf3-9456-4e29-b6c8-aaad902d9d5c\") " pod="openshift-dns/node-resolver-hpn2d"
Jul 28 11:30:50.447901 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.447537    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 11:30:50.447901 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.447636    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 11:30:50.447901 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.447745    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 11:30:50.447901 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.447881    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-psl4d\""
Jul 28 11:30:50.448219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.447917    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 11:30:50.448219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.448018    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 11:30:50.448219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.448102    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 11:30:50.448219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.448140    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 11:30:50.448394 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.448245    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 11:30:50.448394 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.448370    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.450748 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.450727    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:30:50.450840 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:50.450802    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg8rp" podUID="1a04e1e5-57de-4b91-b28e-f4ef1b5e67be"
Jul 28 11:30:50.451180 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.451135    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tj8n2\""
Jul 28 11:30:50.451275 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.451218    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 11:30:50.453060 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.453040    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kgjx5"
Jul 28 11:30:50.455618 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.455598    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l2gf5\""
Jul 28 11:30:50.455816 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.455799    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 11:30:50.455894 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.455825    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 11:30:50.456121 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.456098    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 11:30:50.492260 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.492237    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8cp76"
Jul 28 11:30:50.503243 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.503222    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8cp76"
Jul 28 11:30:50.536735 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.536704    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 11:30:50.545006 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.544955    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal" event={"ID":"efb3d0a42d97d596842f3e8617f33a18","Type":"ContainerStarted","Data":"bfaf60ddc60cd4018d687c3aaa5e1ac6d641b7f096df1ca1191950173e0b7dbc"}
Jul 28 11:30:50.546299 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.546266    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal" event={"ID":"8a2ad931bbcb3d6172f2d426a64e2848","Type":"ContainerStarted","Data":"447d4c8171753d6fc55c1555e260f95269a7a2f1500b4f75aa7c8f18d534d3ca"}
Jul 28 11:30:50.548531 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.548510    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-modprobe-d\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.548644 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.548536    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a930d1a2-a540-40bf-bcb9-f4e24dd760af-tmp\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.548644 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.548554    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e552bdf3-9456-4e29-b6c8-aaad902d9d5c-tmp-dir\") pod \"node-resolver-hpn2d\" (UID: \"e552bdf3-9456-4e29-b6c8-aaad902d9d5c\") " pod="openshift-dns/node-resolver-hpn2d"
Jul 28 11:30:50.548644 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.548611    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-kubelet\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.548644 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.548630    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-ovnkube-script-lib\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.548824 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.548655    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-system-cni-dir\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.548824 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.548678    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6xbs\" (UniqueName: \"kubernetes.io/projected/a930d1a2-a540-40bf-bcb9-f4e24dd760af-kube-api-access-q6xbs\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.548824 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.548721    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67mhn\" (UniqueName: \"kubernetes.io/projected/e552bdf3-9456-4e29-b6c8-aaad902d9d5c-kube-api-access-67mhn\") pod \"node-resolver-hpn2d\" (UID: \"e552bdf3-9456-4e29-b6c8-aaad902d9d5c\") " pod="openshift-dns/node-resolver-hpn2d"
Jul 28 11:30:50.548824 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.548742    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-cni-netd\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.548824 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.548777    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-modprobe-d\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.548824 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.548795    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-cnibin\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.549083 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.548954    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e552bdf3-9456-4e29-b6c8-aaad902d9d5c-tmp-dir\") pod \"node-resolver-hpn2d\" (UID: \"e552bdf3-9456-4e29-b6c8-aaad902d9d5c\") " pod="openshift-dns/node-resolver-hpn2d"
Jul 28 11:30:50.549365 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.548914    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-host-var-lib-kubelet\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.549434 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549381    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret\") pod \"global-pull-secret-syncer-fg8rp\" (UID: \"1a04e1e5-57de-4b91-b28e-f4ef1b5e67be\") " pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:30:50.549434 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549417    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-cni-binary-copy\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.549531 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549443    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndmqc\" (UniqueName: \"kubernetes.io/projected/85d71f38-a39a-4a1c-90dc-f5b941567052-kube-api-access-ndmqc\") pod \"network-metrics-daemon-ss7h8\" (UID: \"85d71f38-a39a-4a1c-90dc-f5b941567052\") " pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:30:50.549531 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549470    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e552bdf3-9456-4e29-b6c8-aaad902d9d5c-hosts-file\") pod \"node-resolver-hpn2d\" (UID: \"e552bdf3-9456-4e29-b6c8-aaad902d9d5c\") " pod="openshift-dns/node-resolver-hpn2d"
Jul 28 11:30:50.549531 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549501    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.549531 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549526    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-etc-selinux\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.549715 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549551    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5d56df58-94ae-48e8-85d5-0c2556bb4cc4-iptables-alerter-script\") pod \"iptables-alerter-kgjx5\" (UID: \"5d56df58-94ae-48e8-85d5-0c2556bb4cc4\") " pod="openshift-network-operator/iptables-alerter-kgjx5"
Jul 28 11:30:50.549715 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549592    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e552bdf3-9456-4e29-b6c8-aaad902d9d5c-hosts-file\") pod \"node-resolver-hpn2d\" (UID: \"e552bdf3-9456-4e29-b6c8-aaad902d9d5c\") " pod="openshift-dns/node-resolver-hpn2d"
Jul 28 11:30:50.549715 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9af318d1-7622-4574-8c01-ee23cbcf4fdc-serviceca\") pod \"node-ca-pthzf\" (UID: \"9af318d1-7622-4574-8c01-ee23cbcf4fdc\") " pod="openshift-image-registry/node-ca-pthzf"
Jul 28 11:30:50.549715 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549645    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-host\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.549715 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549678    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jqm5\" (UniqueName: \"kubernetes.io/projected/9af318d1-7622-4574-8c01-ee23cbcf4fdc-kube-api-access-9jqm5\") pod \"node-ca-pthzf\" (UID: \"9af318d1-7622-4574-8c01-ee23cbcf4fdc\") " pod="openshift-image-registry/node-ca-pthzf"
Jul 28 11:30:50.549715 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549705    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-systemd-units\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.550064 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549729    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-etc-openvswitch\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.550064 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-cni-bin\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.550064 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549777    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-host\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.550064 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549799    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.550064 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-multus-cni-dir\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.550064 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549885    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-cni-binary-copy\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.550064 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549944    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-node-log\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.550064 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549953    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-cni-binary-copy\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.550064 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549971    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-host-run-netns\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.550064 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.549997    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-host-var-lib-cni-multus\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.550064 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550023    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-etc-kubernetes\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.550064 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550017    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 11:30:50.550064 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550048    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb72k\" (UniqueName: \"kubernetes.io/projected/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-kube-api-access-jb72k\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.550686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550074    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-run\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.550686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550100    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-tuned\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.550686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550123    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-dbus\") pod \"global-pull-secret-syncer-fg8rp\" (UID: \"1a04e1e5-57de-4b91-b28e-f4ef1b5e67be\") " pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:30:50.550686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550148    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.550686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550175    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b0fd488-b3f4-4e45-8832-a74efd910839-agent-certs\") pod \"konnectivity-agent-xwtf2\" (UID: \"0b0fd488-b3f4-4e45-8832-a74efd910839\") " pod="kube-system/konnectivity-agent-xwtf2"
Jul 28 11:30:50.550686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550197    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-run-openvswitch\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.550686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550222    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-run-ovn-kubernetes\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.550686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550255    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-host-run-k8s-cni-cncf-io\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.550686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550281    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-sys\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.550686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550303    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-run-netns\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.550686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550321    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-run\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.550686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550328    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.550686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550353    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-device-dir\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.550686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550377    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbc8b\" (UniqueName: \"kubernetes.io/projected/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-kube-api-access-xbc8b\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.550686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550400    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jjvg\" (UniqueName: \"kubernetes.io/projected/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-kube-api-access-6jjvg\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.550686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550423    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-hostroot\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.550686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550448    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-system-cni-dir\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.551412 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550459    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-sys\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.551412 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-os-release\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.551412 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550498    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.551412 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550524    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-system-cni-dir\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.551412 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550528    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmr7x\" (UniqueName: \"kubernetes.io/projected/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-kube-api-access-wmr7x\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.551412 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550564    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-os-release\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.551412 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550617    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-kubernetes\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.551412 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550648    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-registration-dir\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.551412 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d56df58-94ae-48e8-85d5-0c2556bb4cc4-host-slash\") pod \"iptables-alerter-kgjx5\" (UID: \"5d56df58-94ae-48e8-85d5-0c2556bb4cc4\") " pod="openshift-network-operator/iptables-alerter-kgjx5"
Jul 28 11:30:50.551412 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550691    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t877b\" (UniqueName: \"kubernetes.io/projected/5d56df58-94ae-48e8-85d5-0c2556bb4cc4-kube-api-access-t877b\") pod \"iptables-alerter-kgjx5\" (UID: \"5d56df58-94ae-48e8-85d5-0c2556bb4cc4\") " pod="openshift-network-operator/iptables-alerter-kgjx5"
Jul 28 11:30:50.551412 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550712    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-kubernetes\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.551412 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550716    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-slash\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.551412 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550743    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-cnibin\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.551412 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550764    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b0fd488-b3f4-4e45-8832-a74efd910839-konnectivity-ca\") pod \"konnectivity-agent-xwtf2\" (UID: \"0b0fd488-b3f4-4e45-8832-a74efd910839\") " pod="kube-system/konnectivity-agent-xwtf2"
Jul 28 11:30:50.551412 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550793    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-socket-dir\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.551412 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550801    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.552083 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550833    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-var-lib-openvswitch\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.552083 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550861    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-multus-socket-dir-parent\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.552083 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550838    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-cnibin\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.552083 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550883    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-var-lib-kubelet\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.552083 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550937    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-run-systemd\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.552083 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550962    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-env-overrides\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.552083 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550967    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-var-lib-kubelet\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.552083 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.550994    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-host-var-lib-cni-bin\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.552083 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551018    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-sysctl-conf\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.552083 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9af318d1-7622-4574-8c01-ee23cbcf4fdc-host\") pod \"node-ca-pthzf\" (UID: \"9af318d1-7622-4574-8c01-ee23cbcf4fdc\") " pod="openshift-image-registry/node-ca-pthzf"
Jul 28 11:30:50.552083 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551064    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-log-socket\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.552083 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551083    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-ovnkube-config\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.552083 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551102    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-ovn-node-metrics-cert\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.552083 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551122    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-host-run-multus-certs\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.552083 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551157    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-sysconfig\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.552083 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551164    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-sysctl-conf\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.552083 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551193    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-sysctl-d\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.552773 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551213    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-lib-modules\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.552773 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551226    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-sysconfig\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.552773 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551234    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-sys-fs\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.552773 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551255    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-multus-daemon-config\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.552773 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551277    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88qq\" (UniqueName: \"kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq\") pod \"network-check-target-bcd9g\" (UID: \"35b9e83d-edd8-4afa-9368-a7a490338b85\") " pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:30:50.552773 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551297    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-systemd\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.552773 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551308    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b0fd488-b3f4-4e45-8832-a74efd910839-konnectivity-ca\") pod \"konnectivity-agent-xwtf2\" (UID: \"0b0fd488-b3f4-4e45-8832-a74efd910839\") " pod="kube-system/konnectivity-agent-xwtf2"
Jul 28 11:30:50.552773 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551320    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-run-ovn\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.552773 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551329    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-lib-modules\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.552773 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551350    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-os-release\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.552773 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551324    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-sysctl-d\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.552773 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551475    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-multus-conf-dir\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.552773 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551478    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-systemd\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.552773 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551518    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-kubelet-config\") pod \"global-pull-secret-syncer-fg8rp\" (UID: \"1a04e1e5-57de-4b91-b28e-f4ef1b5e67be\") " pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:30:50.552773 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551561    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.552773 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.551616    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs\") pod \"network-metrics-daemon-ss7h8\" (UID: \"85d71f38-a39a-4a1c-90dc-f5b941567052\") " pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:30:50.552773 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:50.551741    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 11:30:50.553427 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:50.551808    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs podName:85d71f38-a39a-4a1c-90dc-f5b941567052 nodeName:}" failed. No retries permitted until 2026-07-28 11:30:51.051788269 +0000 UTC m=+2.910444267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs") pod "network-metrics-daemon-ss7h8" (UID: "85d71f38-a39a-4a1c-90dc-f5b941567052") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 11:30:50.553427 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.552033    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.554004 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.553982    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a930d1a2-a540-40bf-bcb9-f4e24dd760af-tmp\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.554090 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.554007    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a930d1a2-a540-40bf-bcb9-f4e24dd760af-etc-tuned\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.554205 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.554189    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b0fd488-b3f4-4e45-8832-a74efd910839-agent-certs\") pod \"konnectivity-agent-xwtf2\" (UID: \"0b0fd488-b3f4-4e45-8832-a74efd910839\") " pod="kube-system/konnectivity-agent-xwtf2"
Jul 28 11:30:50.566132 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.566112    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6xbs\" (UniqueName: \"kubernetes.io/projected/a930d1a2-a540-40bf-bcb9-f4e24dd760af-kube-api-access-q6xbs\") pod \"tuned-dffdp\" (UID: \"a930d1a2-a540-40bf-bcb9-f4e24dd760af\") " pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.566729 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:50.566499    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 11:30:50.566729 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:50.566523    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 11:30:50.566729 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:50.566535    2561 projected.go:196] Error preparing data for projected volume kube-api-access-d88qq for pod openshift-network-diagnostics/network-check-target-bcd9g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 11:30:50.566729 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:50.566601    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq podName:35b9e83d-edd8-4afa-9368-a7a490338b85 nodeName:}" failed. No retries permitted until 2026-07-28 11:30:51.066571795 +0000 UTC m=+2.925227772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d88qq" (UniqueName: "kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq") pod "network-check-target-bcd9g" (UID: "35b9e83d-edd8-4afa-9368-a7a490338b85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 11:30:50.568592 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.568553    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmr7x\" (UniqueName: \"kubernetes.io/projected/bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56-kube-api-access-wmr7x\") pod \"multus-additional-cni-plugins-wprpw\" (UID: \"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56\") " pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.568958 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.568772    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndmqc\" (UniqueName: \"kubernetes.io/projected/85d71f38-a39a-4a1c-90dc-f5b941567052-kube-api-access-ndmqc\") pod \"network-metrics-daemon-ss7h8\" (UID: \"85d71f38-a39a-4a1c-90dc-f5b941567052\") " pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:30:50.570119 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.570101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67mhn\" (UniqueName: \"kubernetes.io/projected/e552bdf3-9456-4e29-b6c8-aaad902d9d5c-kube-api-access-67mhn\") pod \"node-resolver-hpn2d\" (UID: \"e552bdf3-9456-4e29-b6c8-aaad902d9d5c\") " pod="openshift-dns/node-resolver-hpn2d"
Jul 28 11:30:50.652187 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652112    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-host-run-netns\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.652187 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-host-var-lib-cni-multus\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.652187 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652160    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-etc-kubernetes\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.652187 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652178    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb72k\" (UniqueName: \"kubernetes.io/projected/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-kube-api-access-jb72k\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.652466 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652226    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-host-run-netns\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.652466 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652237    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-etc-kubernetes\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.652466 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652248    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-dbus\") pod \"global-pull-secret-syncer-fg8rp\" (UID: \"1a04e1e5-57de-4b91-b28e-f4ef1b5e67be\") " pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:30:50.652466 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652281    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-run-openvswitch\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.652466 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652297    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-run-ovn-kubernetes\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.652466 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652313    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-host-run-k8s-cni-cncf-io\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.652466 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652331    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-run-netns\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.652466 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652357    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-device-dir\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.652466 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652356    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-run-openvswitch\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.652466 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652231    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-host-var-lib-cni-multus\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.652466 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652383    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbc8b\" (UniqueName: \"kubernetes.io/projected/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-kube-api-access-xbc8b\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.652466 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652388    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-dbus\") pod \"global-pull-secret-syncer-fg8rp\" (UID: \"1a04e1e5-57de-4b91-b28e-f4ef1b5e67be\") " pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:30:50.652466 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652382    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-run-ovn-kubernetes\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.652466 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652419    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jjvg\" (UniqueName: \"kubernetes.io/projected/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-kube-api-access-6jjvg\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.652466 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652433    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-device-dir\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.652466 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652448    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-hostroot\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.652466 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652437    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-run-netns\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.653120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652414    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-host-run-k8s-cni-cncf-io\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.653120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652495    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-hostroot\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.653120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652525    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-registration-dir\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.653120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652551    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d56df58-94ae-48e8-85d5-0c2556bb4cc4-host-slash\") pod \"iptables-alerter-kgjx5\" (UID: \"5d56df58-94ae-48e8-85d5-0c2556bb4cc4\") " pod="openshift-network-operator/iptables-alerter-kgjx5"
Jul 28 11:30:50.653120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t877b\" (UniqueName: \"kubernetes.io/projected/5d56df58-94ae-48e8-85d5-0c2556bb4cc4-kube-api-access-t877b\") pod \"iptables-alerter-kgjx5\" (UID: \"5d56df58-94ae-48e8-85d5-0c2556bb4cc4\") " pod="openshift-network-operator/iptables-alerter-kgjx5"
Jul 28 11:30:50.653120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652598    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-registration-dir\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.653120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652618    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-slash\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.653120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652641    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-slash\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.653120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652619    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d56df58-94ae-48e8-85d5-0c2556bb4cc4-host-slash\") pod \"iptables-alerter-kgjx5\" (UID: \"5d56df58-94ae-48e8-85d5-0c2556bb4cc4\") " pod="openshift-network-operator/iptables-alerter-kgjx5"
Jul 28 11:30:50.653120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652643    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-socket-dir\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.653120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652688    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-var-lib-openvswitch\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.653120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652712    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-multus-socket-dir-parent\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.653120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652736    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-run-systemd\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.653120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652755    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-socket-dir\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.653120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652764    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-env-overrides\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.653120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652740    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-var-lib-openvswitch\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.653120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652773    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-multus-socket-dir-parent\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.653862 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652789    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-run-systemd\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.653862 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652825    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-host-var-lib-cni-bin\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.653862 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652881    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-host-var-lib-cni-bin\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.653862 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652950    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9af318d1-7622-4574-8c01-ee23cbcf4fdc-host\") pod \"node-ca-pthzf\" (UID: \"9af318d1-7622-4574-8c01-ee23cbcf4fdc\") " pod="openshift-image-registry/node-ca-pthzf"
Jul 28 11:30:50.653862 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.652990    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-log-socket\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.653862 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653016    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-ovnkube-config\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.653862 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653040    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-ovn-node-metrics-cert\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.653862 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653057    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9af318d1-7622-4574-8c01-ee23cbcf4fdc-host\") pod \"node-ca-pthzf\" (UID: \"9af318d1-7622-4574-8c01-ee23cbcf4fdc\") " pod="openshift-image-registry/node-ca-pthzf"
Jul 28 11:30:50.653862 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653068    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-host-run-multus-certs\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.653862 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653097    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-sys-fs\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.653862 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653119    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-multus-daemon-config\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.653862 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653155    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-host-run-multus-certs\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.653862 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653169    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-run-ovn\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.653862 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653202    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-os-release\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.653862 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653211    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-run-ovn\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.653862 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653228    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-multus-conf-dir\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.653862 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653242    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-sys-fs\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.653862 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653251    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-kubelet-config\") pod \"global-pull-secret-syncer-fg8rp\" (UID: \"1a04e1e5-57de-4b91-b28e-f4ef1b5e67be\") " pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:30:50.654498 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653298    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-kubelet-config\") pod \"global-pull-secret-syncer-fg8rp\" (UID: \"1a04e1e5-57de-4b91-b28e-f4ef1b5e67be\") " pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:30:50.654498 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653303    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-kubelet\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.654498 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653331    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-env-overrides\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.654498 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653344    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-os-release\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.654498 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653334    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-ovnkube-script-lib\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.654498 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653371    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-multus-conf-dir\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.654498 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653067    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-log-socket\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.654498 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653381    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-system-cni-dir\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.654498 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653409    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-cni-netd\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.654498 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653421    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-kubelet\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.654498 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653432    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-cnibin\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.654498 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653470    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-cnibin\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.654498 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-host-var-lib-kubelet\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.654498 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653497    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-system-cni-dir\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.654498 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret\") pod \"global-pull-secret-syncer-fg8rp\" (UID: \"1a04e1e5-57de-4b91-b28e-f4ef1b5e67be\") " pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:30:50.654498 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653510    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-cni-netd\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.654498 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653535    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-host-var-lib-kubelet\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.654498 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653537    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.655134 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653568    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-etc-selinux\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.655134 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653575    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.655134 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653576    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-ovnkube-config\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.655134 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653610    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5d56df58-94ae-48e8-85d5-0c2556bb4cc4-iptables-alerter-script\") pod \"iptables-alerter-kgjx5\" (UID: \"5d56df58-94ae-48e8-85d5-0c2556bb4cc4\") " pod="openshift-network-operator/iptables-alerter-kgjx5"
Jul 28 11:30:50.655134 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9af318d1-7622-4574-8c01-ee23cbcf4fdc-serviceca\") pod \"node-ca-pthzf\" (UID: \"9af318d1-7622-4574-8c01-ee23cbcf4fdc\") " pod="openshift-image-registry/node-ca-pthzf"
Jul 28 11:30:50.655134 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653670    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jqm5\" (UniqueName: \"kubernetes.io/projected/9af318d1-7622-4574-8c01-ee23cbcf4fdc-kube-api-access-9jqm5\") pod \"node-ca-pthzf\" (UID: \"9af318d1-7622-4574-8c01-ee23cbcf4fdc\") " pod="openshift-image-registry/node-ca-pthzf"
Jul 28 11:30:50.655134 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653692    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-systemd-units\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.655134 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:50.653707    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 11:30:50.655134 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653715    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-etc-openvswitch\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.655134 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653711    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-multus-daemon-config\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.655134 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653746    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-systemd-units\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.655134 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653708    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-etc-selinux\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.655134 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:50.653770    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret podName:1a04e1e5-57de-4b91-b28e-f4ef1b5e67be nodeName:}" failed. No retries permitted until 2026-07-28 11:30:51.153751324 +0000 UTC m=+3.012407302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret") pod "global-pull-secret-syncer-fg8rp" (UID: "1a04e1e5-57de-4b91-b28e-f4ef1b5e67be") : object "kube-system"/"original-pull-secret" not registered
Jul 28 11:30:50.655134 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653768    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-etc-openvswitch\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.655134 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653801    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-cni-bin\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.655134 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653821    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-ovnkube-script-lib\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.655134 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653827    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.655786 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653855    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-multus-cni-dir\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.655786 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653863    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.655786 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653879    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-cni-binary-copy\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.655786 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653895    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-host-cni-bin\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.655786 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653902    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-node-log\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.655786 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653921    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-multus-cni-dir\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.655786 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.653933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-node-log\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.655786 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.654132    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9af318d1-7622-4574-8c01-ee23cbcf4fdc-serviceca\") pod \"node-ca-pthzf\" (UID: \"9af318d1-7622-4574-8c01-ee23cbcf4fdc\") " pod="openshift-image-registry/node-ca-pthzf"
Jul 28 11:30:50.655786 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.654252    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5d56df58-94ae-48e8-85d5-0c2556bb4cc4-iptables-alerter-script\") pod \"iptables-alerter-kgjx5\" (UID: \"5d56df58-94ae-48e8-85d5-0c2556bb4cc4\") " pod="openshift-network-operator/iptables-alerter-kgjx5"
Jul 28 11:30:50.655786 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.654327    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-cni-binary-copy\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.656333 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.656306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-ovn-node-metrics-cert\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.670955 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.670929    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbc8b\" (UniqueName: \"kubernetes.io/projected/fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310-kube-api-access-xbc8b\") pod \"aws-ebs-csi-driver-node-vsh96\" (UID: \"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.672280 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.672257    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jjvg\" (UniqueName: \"kubernetes.io/projected/7a2c8b27-c28c-4c58-ad86-917c5e47ebd4-kube-api-access-6jjvg\") pod \"ovnkube-node-f2c2r\" (UID: \"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.673812 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.673794    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jqm5\" (UniqueName: \"kubernetes.io/projected/9af318d1-7622-4574-8c01-ee23cbcf4fdc-kube-api-access-9jqm5\") pod \"node-ca-pthzf\" (UID: \"9af318d1-7622-4574-8c01-ee23cbcf4fdc\") " pod="openshift-image-registry/node-ca-pthzf"
Jul 28 11:30:50.674486 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.674468    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb72k\" (UniqueName: \"kubernetes.io/projected/92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3-kube-api-access-jb72k\") pod \"multus-ht9lx\" (UID: \"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3\") " pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:50.675307 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.675287    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t877b\" (UniqueName: \"kubernetes.io/projected/5d56df58-94ae-48e8-85d5-0c2556bb4cc4-kube-api-access-t877b\") pod \"iptables-alerter-kgjx5\" (UID: \"5d56df58-94ae-48e8-85d5-0c2556bb4cc4\") " pod="openshift-network-operator/iptables-alerter-kgjx5"
Jul 28 11:30:50.744042 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.744021    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dffdp"
Jul 28 11:30:50.753297 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.753280    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hpn2d"
Jul 28 11:30:50.761295 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.761280    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xwtf2"
Jul 28 11:30:50.769943 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.769922    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kgjx5"
Jul 28 11:30:50.777419 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.777402    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wprpw"
Jul 28 11:30:50.783905 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.783889    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pthzf"
Jul 28 11:30:50.789630 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.789601    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:30:50.798370 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.798351    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96"
Jul 28 11:30:50.803092 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:50.803076    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ht9lx"
Jul 28 11:30:51.056791 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:51.056726    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs\") pod \"network-metrics-daemon-ss7h8\" (UID: \"85d71f38-a39a-4a1c-90dc-f5b941567052\") " pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:30:51.056916 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:51.056842    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 11:30:51.056916 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:51.056896    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs podName:85d71f38-a39a-4a1c-90dc-f5b941567052 nodeName:}" failed. No retries permitted until 2026-07-28 11:30:52.05688223 +0000 UTC m=+3.915538208 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs") pod "network-metrics-daemon-ss7h8" (UID: "85d71f38-a39a-4a1c-90dc-f5b941567052") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 11:30:51.157831 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:51.157799    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret\") pod \"global-pull-secret-syncer-fg8rp\" (UID: \"1a04e1e5-57de-4b91-b28e-f4ef1b5e67be\") " pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:30:51.157970 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:51.157872    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88qq\" (UniqueName: \"kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq\") pod \"network-check-target-bcd9g\" (UID: \"35b9e83d-edd8-4afa-9368-a7a490338b85\") " pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:30:51.158019 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:51.157971    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 11:30:51.158019 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:51.157992    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 11:30:51.158019 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:51.158009    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 11:30:51.158019 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:51.158020    2561 projected.go:196] Error preparing data for projected volume kube-api-access-d88qq for pod openshift-network-diagnostics/network-check-target-bcd9g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 11:30:51.158184 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:51.158047    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret podName:1a04e1e5-57de-4b91-b28e-f4ef1b5e67be nodeName:}" failed. No retries permitted until 2026-07-28 11:30:52.158027358 +0000 UTC m=+4.016683339 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret") pod "global-pull-secret-syncer-fg8rp" (UID: "1a04e1e5-57de-4b91-b28e-f4ef1b5e67be") : object "kube-system"/"original-pull-secret" not registered
Jul 28 11:30:51.158184 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:51.158073    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq podName:35b9e83d-edd8-4afa-9368-a7a490338b85 nodeName:}" failed. No retries permitted until 2026-07-28 11:30:52.158057545 +0000 UTC m=+4.016713524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d88qq" (UniqueName: "kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq") pod "network-check-target-bcd9g" (UID: "35b9e83d-edd8-4afa-9368-a7a490338b85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 11:30:51.439553 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:51.439523    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode552bdf3_9456_4e29_b6c8_aaad902d9d5c.slice/crio-8a525761674894cd956927ae40d7fd5a7600c8547ae072fdc66d33c92a0b52d9 WatchSource:0}: Error finding container 8a525761674894cd956927ae40d7fd5a7600c8547ae072fdc66d33c92a0b52d9: Status 404 returned error can't find the container with id 8a525761674894cd956927ae40d7fd5a7600c8547ae072fdc66d33c92a0b52d9
Jul 28 11:30:51.440421 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:51.440371    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbcc1fa7_2cee_4e8b_a81a_8f4959d4b310.slice/crio-c70b4996e8af168b017832c972a19a4575f2c4868662e05af676c72d7985214b WatchSource:0}: Error finding container c70b4996e8af168b017832c972a19a4575f2c4868662e05af676c72d7985214b: Status 404 returned error can't find the container with id c70b4996e8af168b017832c972a19a4575f2c4868662e05af676c72d7985214b
Jul 28 11:30:51.441511 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:51.441487    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d56df58_94ae_48e8_85d5_0c2556bb4cc4.slice/crio-541dc52b6fbef34df71238fdac2b30fa82252583c73d51420ab18ca5f4d46f8b WatchSource:0}: Error finding container 541dc52b6fbef34df71238fdac2b30fa82252583c73d51420ab18ca5f4d46f8b: Status 404 returned error can't find the container with id 541dc52b6fbef34df71238fdac2b30fa82252583c73d51420ab18ca5f4d46f8b
Jul 28 11:30:51.445861 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:51.445842    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0fd488_b3f4_4e45_8832_a74efd910839.slice/crio-cd047cdc18394a797019441298a49afbf3c4bd93d9e4325bb82a9a0c65b956a2 WatchSource:0}: Error finding container cd047cdc18394a797019441298a49afbf3c4bd93d9e4325bb82a9a0c65b956a2: Status 404 returned error can't find the container with id cd047cdc18394a797019441298a49afbf3c4bd93d9e4325bb82a9a0c65b956a2
Jul 28 11:30:51.446840 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:51.446792    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda930d1a2_a540_40bf_bcb9_f4e24dd760af.slice/crio-aa3a3a72f1cf501101116fb97bb8884fe3c3d5fc12cfc696950b9df2d481882e WatchSource:0}: Error finding container aa3a3a72f1cf501101116fb97bb8884fe3c3d5fc12cfc696950b9df2d481882e: Status 404 returned error can't find the container with id aa3a3a72f1cf501101116fb97bb8884fe3c3d5fc12cfc696950b9df2d481882e
Jul 28 11:30:51.447779 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:51.447741    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af318d1_7622_4574_8c01_ee23cbcf4fdc.slice/crio-c2bf204ba17bf353a369be4b5dab52ae5a8ef899714d4e541b966af87daa5008 WatchSource:0}: Error finding container c2bf204ba17bf353a369be4b5dab52ae5a8ef899714d4e541b966af87daa5008: Status 404 returned error can't find the container with id c2bf204ba17bf353a369be4b5dab52ae5a8ef899714d4e541b966af87daa5008
Jul 28 11:30:51.449014 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:51.448484    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0d1c3d_b6b1_4a4a_baac_bdddc95a5b56.slice/crio-02cfea18aabb4e9c0695aa11948af79c172bf8fb7f521bc3926ad45094a6c4a7 WatchSource:0}: Error finding container 02cfea18aabb4e9c0695aa11948af79c172bf8fb7f521bc3926ad45094a6c4a7: Status 404 returned error can't find the container with id 02cfea18aabb4e9c0695aa11948af79c172bf8fb7f521bc3926ad45094a6c4a7
Jul 28 11:30:51.449233 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:51.449213    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2c8b27_c28c_4c58_ad86_917c5e47ebd4.slice/crio-e2052b94909ca3b89bf98354881685331c84ea1b7922a903da65c599d31a0233 WatchSource:0}: Error finding container e2052b94909ca3b89bf98354881685331c84ea1b7922a903da65c599d31a0233: Status 404 returned error can't find the container with id e2052b94909ca3b89bf98354881685331c84ea1b7922a903da65c599d31a0233
Jul 28 11:30:51.451979 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:30:51.451955    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92af682d_4a4b_4cfc_a4bb_e4f24df0d7d3.slice/crio-e4705bc06b158a002425a960d40aebedfc13538a099392273e231159e4aa47e8 WatchSource:0}: Error finding container e4705bc06b158a002425a960d40aebedfc13538a099392273e231159e4aa47e8: Status 404 returned error can't find the container with id e4705bc06b158a002425a960d40aebedfc13538a099392273e231159e4aa47e8
Jul 28 11:30:51.506149 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:51.506100    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 11:25:50 +0000 UTC" deadline="2028-01-29 05:04:55.276160202 +0000 UTC"
Jul 28 11:30:51.506233 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:51.506152    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13193h34m3.770011946s"
Jul 28 11:30:51.550904 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:51.550879    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96" event={"ID":"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310","Type":"ContainerStarted","Data":"c70b4996e8af168b017832c972a19a4575f2c4868662e05af676c72d7985214b"}
Jul 28 11:30:51.554778 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:51.554757    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal" event={"ID":"8a2ad931bbcb3d6172f2d426a64e2848","Type":"ContainerStarted","Data":"11882365b0cf756cde77e694f03d193353b362a45d32ff0a0ca5b7cdc7c7dfee"}
Jul 28 11:30:51.556823 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:51.556803    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wprpw" event={"ID":"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56","Type":"ContainerStarted","Data":"02cfea18aabb4e9c0695aa11948af79c172bf8fb7f521bc3926ad45094a6c4a7"}
Jul 28 11:30:51.561029 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:51.561009    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kgjx5" event={"ID":"5d56df58-94ae-48e8-85d5-0c2556bb4cc4","Type":"ContainerStarted","Data":"541dc52b6fbef34df71238fdac2b30fa82252583c73d51420ab18ca5f4d46f8b"}
Jul 28 11:30:51.563312 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:51.563292    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hpn2d" event={"ID":"e552bdf3-9456-4e29-b6c8-aaad902d9d5c","Type":"ContainerStarted","Data":"8a525761674894cd956927ae40d7fd5a7600c8547ae072fdc66d33c92a0b52d9"}
Jul 28 11:30:51.564446 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:51.564424    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ht9lx" event={"ID":"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3","Type":"ContainerStarted","Data":"e4705bc06b158a002425a960d40aebedfc13538a099392273e231159e4aa47e8"}
Jul 28 11:30:51.565615 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:51.565592    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r" event={"ID":"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4","Type":"ContainerStarted","Data":"e2052b94909ca3b89bf98354881685331c84ea1b7922a903da65c599d31a0233"}
Jul 28 11:30:51.566562 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:51.566539    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pthzf" event={"ID":"9af318d1-7622-4574-8c01-ee23cbcf4fdc","Type":"ContainerStarted","Data":"c2bf204ba17bf353a369be4b5dab52ae5a8ef899714d4e541b966af87daa5008"}
Jul 28 11:30:51.567938 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:51.567918    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dffdp" event={"ID":"a930d1a2-a540-40bf-bcb9-f4e24dd760af","Type":"ContainerStarted","Data":"aa3a3a72f1cf501101116fb97bb8884fe3c3d5fc12cfc696950b9df2d481882e"}
Jul 28 11:30:51.568900 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:51.568877    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xwtf2" event={"ID":"0b0fd488-b3f4-4e45-8832-a74efd910839","Type":"ContainerStarted","Data":"cd047cdc18394a797019441298a49afbf3c4bd93d9e4325bb82a9a0c65b956a2"}
Jul 28 11:30:51.571161 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:51.571123    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal" podStartSLOduration=2.571111823 podStartE2EDuration="2.571111823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 11:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:30:51.570616209 +0000 UTC m=+3.429272205" watchObservedRunningTime="2026-07-28 11:30:51.571111823 +0000 UTC m=+3.429767822"
Jul 28 11:30:52.013441 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:52.013407    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 11:30:52.065189 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:52.065140    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs\") pod \"network-metrics-daemon-ss7h8\" (UID: \"85d71f38-a39a-4a1c-90dc-f5b941567052\") " pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:30:52.065357 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:52.065307    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 11:30:52.065414 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:52.065373    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs podName:85d71f38-a39a-4a1c-90dc-f5b941567052 nodeName:}" failed. No retries permitted until 2026-07-28 11:30:54.065354003 +0000 UTC m=+5.924010008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs") pod "network-metrics-daemon-ss7h8" (UID: "85d71f38-a39a-4a1c-90dc-f5b941567052") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 11:30:52.166038 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:52.166000    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88qq\" (UniqueName: \"kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq\") pod \"network-check-target-bcd9g\" (UID: \"35b9e83d-edd8-4afa-9368-a7a490338b85\") " pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:30:52.166223 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:52.166075    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret\") pod \"global-pull-secret-syncer-fg8rp\" (UID: \"1a04e1e5-57de-4b91-b28e-f4ef1b5e67be\") " pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:30:52.166223 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:52.166217    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 11:30:52.166336 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:52.166283    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret podName:1a04e1e5-57de-4b91-b28e-f4ef1b5e67be nodeName:}" failed. No retries permitted until 2026-07-28 11:30:54.166263598 +0000 UTC m=+6.024919581 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret") pod "global-pull-secret-syncer-fg8rp" (UID: "1a04e1e5-57de-4b91-b28e-f4ef1b5e67be") : object "kube-system"/"original-pull-secret" not registered
Jul 28 11:30:52.166809 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:52.166781    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 11:30:52.166899 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:52.166814    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 11:30:52.166899 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:52.166828    2561 projected.go:196] Error preparing data for projected volume kube-api-access-d88qq for pod openshift-network-diagnostics/network-check-target-bcd9g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 11:30:52.166899 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:52.166871    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq podName:35b9e83d-edd8-4afa-9368-a7a490338b85 nodeName:}" failed. No retries permitted until 2026-07-28 11:30:54.166856945 +0000 UTC m=+6.025512925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d88qq" (UniqueName: "kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq") pod "network-check-target-bcd9g" (UID: "35b9e83d-edd8-4afa-9368-a7a490338b85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 11:30:52.506816 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:52.506729    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 11:25:50 +0000 UTC" deadline="2028-03-07 21:52:44.234012971 +0000 UTC"
Jul 28 11:30:52.506816 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:52.506769    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14122h21m51.727247968s"
Jul 28 11:30:52.541016 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:52.540989    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:30:52.541173 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:52.541149    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ss7h8" podUID="85d71f38-a39a-4a1c-90dc-f5b941567052"
Jul 28 11:30:52.541569 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:52.541544    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:30:52.541671 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:52.541652    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcd9g" podUID="35b9e83d-edd8-4afa-9368-a7a490338b85"
Jul 28 11:30:52.541732 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:52.541722    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:30:52.541808 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:52.541789    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg8rp" podUID="1a04e1e5-57de-4b91-b28e-f4ef1b5e67be"
Jul 28 11:30:52.581396 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:52.581346    2561 generic.go:356] "Generic (PLEG): container finished" podID="efb3d0a42d97d596842f3e8617f33a18" containerID="88f5883fca9e9ed87aab6eb8d18969a17dd91b0dd57eb4b4f3d85f9177a98995" exitCode=0
Jul 28 11:30:52.582347 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:52.582323    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal" event={"ID":"efb3d0a42d97d596842f3e8617f33a18","Type":"ContainerDied","Data":"88f5883fca9e9ed87aab6eb8d18969a17dd91b0dd57eb4b4f3d85f9177a98995"}
Jul 28 11:30:53.611117 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:53.610337    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal" event={"ID":"efb3d0a42d97d596842f3e8617f33a18","Type":"ContainerStarted","Data":"f14824e0a19e88b2214c3c231cbbdb52a1ca43d0977119f06ef44e09c651216a"}
Jul 28 11:30:54.083922 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:54.083210    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs\") pod \"network-metrics-daemon-ss7h8\" (UID: \"85d71f38-a39a-4a1c-90dc-f5b941567052\") " pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:30:54.083922 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:54.083403    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 11:30:54.083922 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:54.083468    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs podName:85d71f38-a39a-4a1c-90dc-f5b941567052 nodeName:}" failed. No retries permitted until 2026-07-28 11:30:58.083448454 +0000 UTC m=+9.942104455 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs") pod "network-metrics-daemon-ss7h8" (UID: "85d71f38-a39a-4a1c-90dc-f5b941567052") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 11:30:54.184651 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:54.183849    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88qq\" (UniqueName: \"kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq\") pod \"network-check-target-bcd9g\" (UID: \"35b9e83d-edd8-4afa-9368-a7a490338b85\") " pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:30:54.184651 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:54.183917    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret\") pod \"global-pull-secret-syncer-fg8rp\" (UID: \"1a04e1e5-57de-4b91-b28e-f4ef1b5e67be\") " pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:30:54.184651 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:54.184047    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 11:30:54.184651 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:54.184107    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret podName:1a04e1e5-57de-4b91-b28e-f4ef1b5e67be nodeName:}" failed. No retries permitted until 2026-07-28 11:30:58.184089565 +0000 UTC m=+10.042745544 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret") pod "global-pull-secret-syncer-fg8rp" (UID: "1a04e1e5-57de-4b91-b28e-f4ef1b5e67be") : object "kube-system"/"original-pull-secret" not registered
Jul 28 11:30:54.184651 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:54.184516    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 11:30:54.184651 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:54.184537    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 11:30:54.184651 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:54.184549    2561 projected.go:196] Error preparing data for projected volume kube-api-access-d88qq for pod openshift-network-diagnostics/network-check-target-bcd9g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 11:30:54.184651 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:54.184607    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq podName:35b9e83d-edd8-4afa-9368-a7a490338b85 nodeName:}" failed. No retries permitted until 2026-07-28 11:30:58.184592377 +0000 UTC m=+10.043248355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d88qq" (UniqueName: "kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq") pod "network-check-target-bcd9g" (UID: "35b9e83d-edd8-4afa-9368-a7a490338b85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 11:30:54.540726 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:54.540652    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:30:54.540884 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:54.540654    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:30:54.540884 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:54.540812    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcd9g" podUID="35b9e83d-edd8-4afa-9368-a7a490338b85"
Jul 28 11:30:54.540997 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:54.540911    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg8rp" podUID="1a04e1e5-57de-4b91-b28e-f4ef1b5e67be"
Jul 28 11:30:54.540997 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:54.540974    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:30:54.541106 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:54.541053    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ss7h8" podUID="85d71f38-a39a-4a1c-90dc-f5b941567052"
Jul 28 11:30:56.540701 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:56.540663    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:30:56.541166 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:56.540822    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ss7h8" podUID="85d71f38-a39a-4a1c-90dc-f5b941567052"
Jul 28 11:30:56.541166 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:56.541029    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:30:56.541166 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:56.541134    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg8rp" podUID="1a04e1e5-57de-4b91-b28e-f4ef1b5e67be"
Jul 28 11:30:56.541331 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:56.541209    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:30:56.541331 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:56.541295    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcd9g" podUID="35b9e83d-edd8-4afa-9368-a7a490338b85"
Jul 28 11:30:58.120601 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:58.120440    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs\") pod \"network-metrics-daemon-ss7h8\" (UID: \"85d71f38-a39a-4a1c-90dc-f5b941567052\") " pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:30:58.121087 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:58.120606    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 11:30:58.121087 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:58.120682    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs podName:85d71f38-a39a-4a1c-90dc-f5b941567052 nodeName:}" failed. No retries permitted until 2026-07-28 11:31:06.120661325 +0000 UTC m=+17.979317301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs") pod "network-metrics-daemon-ss7h8" (UID: "85d71f38-a39a-4a1c-90dc-f5b941567052") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 11:30:58.221109 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:58.221079    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88qq\" (UniqueName: \"kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq\") pod \"network-check-target-bcd9g\" (UID: \"35b9e83d-edd8-4afa-9368-a7a490338b85\") " pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:30:58.221298 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:58.221142    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret\") pod \"global-pull-secret-syncer-fg8rp\" (UID: \"1a04e1e5-57de-4b91-b28e-f4ef1b5e67be\") " pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:30:58.221298 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:58.221240    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 11:30:58.221298 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:58.221299    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret podName:1a04e1e5-57de-4b91-b28e-f4ef1b5e67be nodeName:}" failed. No retries permitted until 2026-07-28 11:31:06.2212704 +0000 UTC m=+18.079926377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret") pod "global-pull-secret-syncer-fg8rp" (UID: "1a04e1e5-57de-4b91-b28e-f4ef1b5e67be") : object "kube-system"/"original-pull-secret" not registered
Jul 28 11:30:58.221702 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:58.221687    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 11:30:58.221755 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:58.221705    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 11:30:58.221755 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:58.221715    2561 projected.go:196] Error preparing data for projected volume kube-api-access-d88qq for pod openshift-network-diagnostics/network-check-target-bcd9g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 11:30:58.221755 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:58.221751    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq podName:35b9e83d-edd8-4afa-9368-a7a490338b85 nodeName:}" failed. No retries permitted until 2026-07-28 11:31:06.221740941 +0000 UTC m=+18.080396919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d88qq" (UniqueName: "kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq") pod "network-check-target-bcd9g" (UID: "35b9e83d-edd8-4afa-9368-a7a490338b85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 11:30:58.542648 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:58.541893    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:30:58.542648 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:58.542001    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcd9g" podUID="35b9e83d-edd8-4afa-9368-a7a490338b85"
Jul 28 11:30:58.542648 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:58.542357    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:30:58.542648 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:58.542456    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg8rp" podUID="1a04e1e5-57de-4b91-b28e-f4ef1b5e67be"
Jul 28 11:30:58.542648 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:30:58.542511    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:30:58.542648 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:30:58.542609    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ss7h8" podUID="85d71f38-a39a-4a1c-90dc-f5b941567052"
Jul 28 11:31:00.541257 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:00.541218    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:31:00.541257 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:00.541237    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:31:00.541736 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:00.541370    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ss7h8" podUID="85d71f38-a39a-4a1c-90dc-f5b941567052"
Jul 28 11:31:00.541736 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:00.541723    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:31:00.541850 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:00.541828    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcd9g" podUID="35b9e83d-edd8-4afa-9368-a7a490338b85"
Jul 28 11:31:00.541926 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:00.541905    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg8rp" podUID="1a04e1e5-57de-4b91-b28e-f4ef1b5e67be"
Jul 28 11:31:02.540489 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:02.540411    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:31:02.540943 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:02.540416    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:31:02.540943 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:02.540534    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg8rp" podUID="1a04e1e5-57de-4b91-b28e-f4ef1b5e67be"
Jul 28 11:31:02.540943 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:02.540644    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ss7h8" podUID="85d71f38-a39a-4a1c-90dc-f5b941567052"
Jul 28 11:31:02.540943 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:02.540414    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:31:02.540943 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:02.540740    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcd9g" podUID="35b9e83d-edd8-4afa-9368-a7a490338b85"
Jul 28 11:31:04.541368 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:04.541332    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:31:04.541368 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:04.541373    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:31:04.541911 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:04.541332    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:31:04.541911 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:04.541463    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ss7h8" podUID="85d71f38-a39a-4a1c-90dc-f5b941567052"
Jul 28 11:31:04.541911 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:04.541603    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcd9g" podUID="35b9e83d-edd8-4afa-9368-a7a490338b85"
Jul 28 11:31:04.541911 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:04.541687    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg8rp" podUID="1a04e1e5-57de-4b91-b28e-f4ef1b5e67be"
Jul 28 11:31:06.184548 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:06.184522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs\") pod \"network-metrics-daemon-ss7h8\" (UID: \"85d71f38-a39a-4a1c-90dc-f5b941567052\") " pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:31:06.184888 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:06.184668    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 11:31:06.184888 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:06.184719    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs podName:85d71f38-a39a-4a1c-90dc-f5b941567052 nodeName:}" failed. No retries permitted until 2026-07-28 11:31:22.184705219 +0000 UTC m=+34.043361197 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs") pod "network-metrics-daemon-ss7h8" (UID: "85d71f38-a39a-4a1c-90dc-f5b941567052") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 11:31:06.285630 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:06.285589    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88qq\" (UniqueName: \"kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq\") pod \"network-check-target-bcd9g\" (UID: \"35b9e83d-edd8-4afa-9368-a7a490338b85\") " pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:31:06.285780 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:06.285664    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret\") pod \"global-pull-secret-syncer-fg8rp\" (UID: \"1a04e1e5-57de-4b91-b28e-f4ef1b5e67be\") " pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:31:06.285780 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:06.285735    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 11:31:06.285780 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:06.285745    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 11:31:06.286000 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:06.285827    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret podName:1a04e1e5-57de-4b91-b28e-f4ef1b5e67be nodeName:}" failed. No retries permitted until 2026-07-28 11:31:22.285813156 +0000 UTC m=+34.144469133 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret") pod "global-pull-secret-syncer-fg8rp" (UID: "1a04e1e5-57de-4b91-b28e-f4ef1b5e67be") : object "kube-system"/"original-pull-secret" not registered
Jul 28 11:31:06.286000 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:06.285760    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 11:31:06.286000 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:06.285881    2561 projected.go:196] Error preparing data for projected volume kube-api-access-d88qq for pod openshift-network-diagnostics/network-check-target-bcd9g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 11:31:06.286000 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:06.285938    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq podName:35b9e83d-edd8-4afa-9368-a7a490338b85 nodeName:}" failed. No retries permitted until 2026-07-28 11:31:22.285921203 +0000 UTC m=+34.144577186 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d88qq" (UniqueName: "kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq") pod "network-check-target-bcd9g" (UID: "35b9e83d-edd8-4afa-9368-a7a490338b85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 11:31:06.541285 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:06.541206    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:31:06.541285 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:06.541234    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:31:06.541491 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:06.541324    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcd9g" podUID="35b9e83d-edd8-4afa-9368-a7a490338b85"
Jul 28 11:31:06.541491 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:06.541391    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg8rp" podUID="1a04e1e5-57de-4b91-b28e-f4ef1b5e67be"
Jul 28 11:31:06.541491 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:06.541418    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:31:06.541660 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:06.541519    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ss7h8" podUID="85d71f38-a39a-4a1c-90dc-f5b941567052"
Jul 28 11:31:08.546267 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:08.546236    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:31:08.546267 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:08.546258    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:31:08.546788 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:08.546236    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:31:08.546788 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:08.546347    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg8rp" podUID="1a04e1e5-57de-4b91-b28e-f4ef1b5e67be"
Jul 28 11:31:08.546788 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:08.546457    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ss7h8" podUID="85d71f38-a39a-4a1c-90dc-f5b941567052"
Jul 28 11:31:08.546788 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:08.546512    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcd9g" podUID="35b9e83d-edd8-4afa-9368-a7a490338b85"
Jul 28 11:31:10.540952 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.540522    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:31:10.541824 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.540528    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:31:10.541824 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:10.541051    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg8rp" podUID="1a04e1e5-57de-4b91-b28e-f4ef1b5e67be"
Jul 28 11:31:10.541824 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:10.541155    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ss7h8" podUID="85d71f38-a39a-4a1c-90dc-f5b941567052"
Jul 28 11:31:10.541824 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.540589    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:31:10.541824 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:10.541232    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcd9g" podUID="35b9e83d-edd8-4afa-9368-a7a490338b85"
Jul 28 11:31:10.645479 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.645451    2561 generic.go:356] "Generic (PLEG): container finished" podID="bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56" containerID="7aed7d1f3acc5bc45abca3aa17c92eaf943df144a44aecc68e882e99bfbfa45a" exitCode=0
Jul 28 11:31:10.645605 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.645538    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wprpw" event={"ID":"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56","Type":"ContainerDied","Data":"7aed7d1f3acc5bc45abca3aa17c92eaf943df144a44aecc68e882e99bfbfa45a"}
Jul 28 11:31:10.646919 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.646888    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hpn2d" event={"ID":"e552bdf3-9456-4e29-b6c8-aaad902d9d5c","Type":"ContainerStarted","Data":"76835aee4a7fdc046c57950411c748e8d742fb83d8dff06c73ff53009824ed5b"}
Jul 28 11:31:10.648287 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.648267    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ht9lx" event={"ID":"92af682d-4a4b-4cfc-a4bb-e4f24df0d7d3","Type":"ContainerStarted","Data":"3c49a8c46c4e4dcc8c68ccb13bf351d53169a9dccd043af44f0c76a222073a6a"}
Jul 28 11:31:10.651173 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.651148    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r" event={"ID":"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4","Type":"ContainerStarted","Data":"872a9c89f4d42228a207e53e9b239e2aa7fb389a12527152f7e904e6706ad578"}
Jul 28 11:31:10.651248 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.651207    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r" event={"ID":"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4","Type":"ContainerStarted","Data":"b616a494e983d4c850465bee358c875b36385790f49c84229cbb4da8ee5fb324"}
Jul 28 11:31:10.651248 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.651238    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r" event={"ID":"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4","Type":"ContainerStarted","Data":"241cc8bbb0bfab261d01bd8683de528f73ad116698e25cff4b0a8a7957905f17"}
Jul 28 11:31:10.651336 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.651271    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r" event={"ID":"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4","Type":"ContainerStarted","Data":"94d4f7f8df09c70c90095a95e33bf6028c4b50665a2d132f5882724ff58cc28c"}
Jul 28 11:31:10.651336 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.651308    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r" event={"ID":"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4","Type":"ContainerStarted","Data":"6ef3fa3c174bcf248ae6c50ac80ba233139d5352ac42966bcd52b0a060ef75aa"}
Jul 28 11:31:10.651393 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.651353    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r" event={"ID":"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4","Type":"ContainerStarted","Data":"b5e6ab916638195e3a46e261375a89a4ecd03bfc404333443d578eaee69eee73"}
Jul 28 11:31:10.652511 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.652490    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pthzf" event={"ID":"9af318d1-7622-4574-8c01-ee23cbcf4fdc","Type":"ContainerStarted","Data":"a63d43006a71ea8ce6f4e0ebdd5b4dd9a398c5682e46e0a338a9b4c21bf6b496"}
Jul 28 11:31:10.653990 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.653971    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dffdp" event={"ID":"a930d1a2-a540-40bf-bcb9-f4e24dd760af","Type":"ContainerStarted","Data":"6a3e428f04d595796c2758db056e8b5b030d76c5efc6089c7db81f36b9a2e1e9"}
Jul 28 11:31:10.655257 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.655237    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xwtf2" event={"ID":"0b0fd488-b3f4-4e45-8832-a74efd910839","Type":"ContainerStarted","Data":"2030800ab85ae9d9c849e4d9df01636a40c34338439d9c50eb6acfe3d3d8b29e"}
Jul 28 11:31:10.656701 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.656679    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96" event={"ID":"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310","Type":"ContainerStarted","Data":"a058d690e75255540822f34f2d7688bfc4c4b659ca97a51e34059f1c5da6f906"}
Jul 28 11:31:10.706816 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.706755    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal" podStartSLOduration=21.706735783 podStartE2EDuration="21.706735783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 11:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:30:53.633260549 +0000 UTC m=+5.491916549" watchObservedRunningTime="2026-07-28 11:31:10.706735783 +0000 UTC m=+22.565391784"
Jul 28 11:31:10.761392 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.761338    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xwtf2" podStartSLOduration=8.046387507 podStartE2EDuration="22.761321429s" totalImagesPullingTime="14.714933922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:31:10.748174932 +0000 UTC m=+22.606830932" watchObservedRunningTime="2026-07-28 11:31:10.761321429 +0000 UTC m=+22.619977427"
Jul 28 11:31:10.785349 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.785300    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pthzf" podStartSLOduration=4.400350437 podStartE2EDuration="22.78528489s" totalImagesPullingTime="18.384934453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 11:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:31:10.78493656 +0000 UTC m=+22.643592559" watchObservedRunningTime="2026-07-28 11:31:10.78528489 +0000 UTC m=+22.643940888"
Jul 28 11:31:10.826274 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.826233    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hpn2d" podStartSLOduration=4.437569833 podStartE2EDuration="22.82621784s" totalImagesPullingTime="18.388648007s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 11:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:31:10.82584492 +0000 UTC m=+22.684500918" watchObservedRunningTime="2026-07-28 11:31:10.82621784 +0000 UTC m=+22.684873832"
Jul 28 11:31:10.867698 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:10.867653    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ht9lx" podStartSLOduration=4.427860772 podStartE2EDuration="22.867636763s" totalImagesPullingTime="18.439775991s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 11:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:31:10.866646818 +0000 UTC m=+22.725302817" watchObservedRunningTime="2026-07-28 11:31:10.867636763 +0000 UTC m=+22.726292761"
Jul 28 11:31:11.148663 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:11.148534    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 11:31:11.516446 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:11.516287    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T11:31:11.14855829Z","UUID":"d1177226-8fb2-4d92-88bc-17f74867ca6c","Handler":null,"Name":"","Endpoint":""}
Jul 28 11:31:11.518043 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:11.518019    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 11:31:11.518158 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:11.518063    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 11:31:11.661151 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:11.661109    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96" event={"ID":"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310","Type":"ContainerStarted","Data":"d6904cf1f98f0dfd88d7c2349a53eb3b8bbcb87c7da48b434303f9d3d9801120"}
Jul 28 11:31:11.662932 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:11.662904    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kgjx5" event={"ID":"5d56df58-94ae-48e8-85d5-0c2556bb4cc4","Type":"ContainerStarted","Data":"bc905143d573f74fbc3b56f68e13844726e04bfb510fdf0b66676339e1b0d36d"}
Jul 28 11:31:11.689527 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:11.689430    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kgjx5" podStartSLOduration=5.303469954 podStartE2EDuration="23.689392449s" totalImagesPullingTime="18.385922495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 11:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:31:11.689147085 +0000 UTC m=+23.547803087" watchObservedRunningTime="2026-07-28 11:31:11.689392449 +0000 UTC m=+23.548048449"
Jul 28 11:31:11.689933 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:11.689892    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dffdp" podStartSLOduration=5.287965876 podStartE2EDuration="23.689884021s" totalImagesPullingTime="18.401918145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 11:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:31:10.919730428 +0000 UTC m=+22.778386419" watchObservedRunningTime="2026-07-28 11:31:11.689884021 +0000 UTC m=+23.548540020"
Jul 28 11:31:12.047938 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:12.047902    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xwtf2"
Jul 28 11:31:12.048627 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:12.048611    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xwtf2"
Jul 28 11:31:12.541478 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:12.541448    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:31:12.541478 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:12.541467    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:31:12.541738 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:12.541601    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ss7h8" podUID="85d71f38-a39a-4a1c-90dc-f5b941567052"
Jul 28 11:31:12.541738 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:12.541632    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:31:12.541851 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:12.541713    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcd9g" podUID="35b9e83d-edd8-4afa-9368-a7a490338b85"
Jul 28 11:31:12.541851 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:12.541813    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg8rp" podUID="1a04e1e5-57de-4b91-b28e-f4ef1b5e67be"
Jul 28 11:31:12.668057 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:12.667961    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96" event={"ID":"fbcc1fa7-2cee-4e8b-a81a-8f4959d4b310","Type":"ContainerStarted","Data":"13a2661a8dbbf88bafa3eaecc5d8163799e5fd5f17f943a12971a013cd8cc89e"}
Jul 28 11:31:12.668493 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:12.668329    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xwtf2"
Jul 28 11:31:12.668942 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:12.668921    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xwtf2"
Jul 28 11:31:12.738570 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:12.738528    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vsh96" podStartSLOduration=4.236969124 podStartE2EDuration="24.738511794s" totalImagesPullingTime="20.50154267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 11:30:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:31:12.707058498 +0000 UTC m=+24.565714498" watchObservedRunningTime="2026-07-28 11:31:12.738511794 +0000 UTC m=+24.597167787"
Jul 28 11:31:13.674804 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:13.674765    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r" event={"ID":"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4","Type":"ContainerStarted","Data":"c843622f79fee7c3f29ecbceea7d1e31a8e77bf21996cabb4368334a77d8843b"}
Jul 28 11:31:14.540681 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:14.540461    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:31:14.540869 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:14.540470    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:31:14.540869 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:14.540533    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:31:14.540869 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:14.540790    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcd9g" podUID="35b9e83d-edd8-4afa-9368-a7a490338b85"
Jul 28 11:31:14.541041 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:14.540870    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg8rp" podUID="1a04e1e5-57de-4b91-b28e-f4ef1b5e67be"
Jul 28 11:31:14.541041 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:14.540966    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ss7h8" podUID="85d71f38-a39a-4a1c-90dc-f5b941567052"
Jul 28 11:31:15.680428 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:15.680259    2561 generic.go:356] "Generic (PLEG): container finished" podID="bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56" containerID="8a493646064d40a278858fcdb26fc9bb0c469523aba57717f59e2b22e013944c" exitCode=0
Jul 28 11:31:15.681226 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:15.680337    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wprpw" event={"ID":"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56","Type":"ContainerDied","Data":"8a493646064d40a278858fcdb26fc9bb0c469523aba57717f59e2b22e013944c"}
Jul 28 11:31:15.683972 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:15.683949    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r" event={"ID":"7a2c8b27-c28c-4c58-ad86-917c5e47ebd4","Type":"ContainerStarted","Data":"ce3d67faab07623ac2b94417b219afa9a8af4ce9d3384151230b92d93472c134"}
Jul 28 11:31:15.684232 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:15.684214    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:31:15.684312 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:15.684268    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:31:15.684312 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:15.684305    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:31:15.699633 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:15.699614    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:31:15.699748 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:15.699711    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:31:16.540810 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:16.540783    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:31:16.540810 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:16.540783    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:31:16.541013 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:16.540783    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:31:16.541013 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:16.540918    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcd9g" podUID="35b9e83d-edd8-4afa-9368-a7a490338b85"
Jul 28 11:31:16.541013 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:16.540972    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ss7h8" podUID="85d71f38-a39a-4a1c-90dc-f5b941567052"
Jul 28 11:31:16.541134 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:16.541046    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg8rp" podUID="1a04e1e5-57de-4b91-b28e-f4ef1b5e67be"
Jul 28 11:31:17.353792 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:17.353737    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r" podStartSLOduration=10.906350261 podStartE2EDuration="29.353720798s" totalImagesPullingTime="18.447370537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 11:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:31:15.783651177 +0000 UTC m=+27.642307176" watchObservedRunningTime="2026-07-28 11:31:17.353720798 +0000 UTC m=+29.212376849"
Jul 28 11:31:17.354143 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:17.353934    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fg8rp"]
Jul 28 11:31:17.354143 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:17.354071    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:31:17.354218 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:17.354166    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg8rp" podUID="1a04e1e5-57de-4b91-b28e-f4ef1b5e67be"
Jul 28 11:31:17.359978 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:17.359953    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bcd9g"]
Jul 28 11:31:17.360108 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:17.360053    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:31:17.360172 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:17.360145    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcd9g" podUID="35b9e83d-edd8-4afa-9368-a7a490338b85"
Jul 28 11:31:17.375717 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:17.375696    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ss7h8"]
Jul 28 11:31:17.375793 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:17.375782    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:31:17.375960 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:17.375880    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ss7h8" podUID="85d71f38-a39a-4a1c-90dc-f5b941567052"
Jul 28 11:31:17.690890 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:17.690819    2561 generic.go:356] "Generic (PLEG): container finished" podID="bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56" containerID="86c3fbed65f78f990db7fc13440c802eee124ccc3d2cb491ecfb062c3c14601c" exitCode=0
Jul 28 11:31:17.691026 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:17.690900    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wprpw" event={"ID":"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56","Type":"ContainerDied","Data":"86c3fbed65f78f990db7fc13440c802eee124ccc3d2cb491ecfb062c3c14601c"}
Jul 28 11:31:18.541925 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:18.541888    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:31:18.542463 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:18.542425    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcd9g" podUID="35b9e83d-edd8-4afa-9368-a7a490338b85"
Jul 28 11:31:18.695615 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:18.695448    2561 generic.go:356] "Generic (PLEG): container finished" podID="bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56" containerID="354aa27e6145b914f168152d704c4a709cfc7d3be8cc7a44227a645461f4e9fc" exitCode=0
Jul 28 11:31:18.695701 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:18.695525    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wprpw" event={"ID":"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56","Type":"ContainerDied","Data":"354aa27e6145b914f168152d704c4a709cfc7d3be8cc7a44227a645461f4e9fc"}
Jul 28 11:31:19.541105 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:19.541073    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:31:19.541314 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:19.541222    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ss7h8" podUID="85d71f38-a39a-4a1c-90dc-f5b941567052"
Jul 28 11:31:19.541314 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:19.541247    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:31:19.541426 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:19.541358    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg8rp" podUID="1a04e1e5-57de-4b91-b28e-f4ef1b5e67be"
Jul 28 11:31:20.540476 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:20.540445    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:31:20.541143 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:20.540544    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bcd9g" podUID="35b9e83d-edd8-4afa-9368-a7a490338b85"
Jul 28 11:31:21.540664 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:21.540568    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:31:21.541103 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:21.540719    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ss7h8" podUID="85d71f38-a39a-4a1c-90dc-f5b941567052"
Jul 28 11:31:21.541103 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:21.540771    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:31:21.541103 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:21.540873    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg8rp" podUID="1a04e1e5-57de-4b91-b28e-f4ef1b5e67be"
Jul 28 11:31:22.208344 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.208307    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs\") pod \"network-metrics-daemon-ss7h8\" (UID: \"85d71f38-a39a-4a1c-90dc-f5b941567052\") " pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:31:22.208528 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:22.208466    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 11:31:22.208624 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:22.208541    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs podName:85d71f38-a39a-4a1c-90dc-f5b941567052 nodeName:}" failed. No retries permitted until 2026-07-28 11:31:54.20852023 +0000 UTC m=+66.067176207 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs") pod "network-metrics-daemon-ss7h8" (UID: "85d71f38-a39a-4a1c-90dc-f5b941567052") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 11:31:22.309251 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.309209    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88qq\" (UniqueName: \"kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq\") pod \"network-check-target-bcd9g\" (UID: \"35b9e83d-edd8-4afa-9368-a7a490338b85\") " pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:31:22.309405 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.309281    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret\") pod \"global-pull-secret-syncer-fg8rp\" (UID: \"1a04e1e5-57de-4b91-b28e-f4ef1b5e67be\") " pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:31:22.309405 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:22.309399    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 11:31:22.309502 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:22.309406    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 11:31:22.309502 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:22.309430    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 11:31:22.309502 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:22.309442    2561 projected.go:196] Error preparing data for projected volume kube-api-access-d88qq for pod openshift-network-diagnostics/network-check-target-bcd9g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 11:31:22.309502 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:22.309454    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret podName:1a04e1e5-57de-4b91-b28e-f4ef1b5e67be nodeName:}" failed. No retries permitted until 2026-07-28 11:31:54.30943945 +0000 UTC m=+66.168095450 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret") pod "global-pull-secret-syncer-fg8rp" (UID: "1a04e1e5-57de-4b91-b28e-f4ef1b5e67be") : object "kube-system"/"original-pull-secret" not registered
Jul 28 11:31:22.309502 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:22.309493    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq podName:35b9e83d-edd8-4afa-9368-a7a490338b85 nodeName:}" failed. No retries permitted until 2026-07-28 11:31:54.309479029 +0000 UTC m=+66.168135029 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d88qq" (UniqueName: "kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq") pod "network-check-target-bcd9g" (UID: "35b9e83d-edd8-4afa-9368-a7a490338b85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 11:31:22.470063 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.469986    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeReady"
Jul 28 11:31:22.470218 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.470144    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 11:31:22.530978 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.530940    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-f7m7q"]
Jul 28 11:31:22.553266 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.553200    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x4jk8"]
Jul 28 11:31:22.554078 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.554052    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:31:22.556845 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.556819    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l5ffw\""
Jul 28 11:31:22.558005 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.557970    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 11:31:22.558400 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.558234    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 11:31:22.571199 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.571179    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x4jk8"
Jul 28 11:31:22.574164 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.573854    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 11:31:22.574164 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.573858    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zjgd5\""
Jul 28 11:31:22.574164 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.573948    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 11:31:22.574164 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.573993    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 11:31:22.574782 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.574766    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f7m7q"]
Jul 28 11:31:22.574866 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.574849    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x4jk8"]
Jul 28 11:31:22.574929 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.574913    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:31:22.574978 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.574934    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:31:22.578353 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.577911    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 11:31:22.578353 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.578015    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2mvc2\""
Jul 28 11:31:22.578353 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.578071    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 11:31:22.578353 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.578012    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tk4rs\""
Jul 28 11:31:22.578353 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.578326    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 11:31:22.712799 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.712768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlf5q\" (UniqueName: \"kubernetes.io/projected/f978d864-c551-4fcf-8a60-ef424a5556a0-kube-api-access-dlf5q\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:31:22.712799 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.712806    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee38fb9c-adce-4aae-9896-5b124fac12bf-cert\") pod \"ingress-canary-x4jk8\" (UID: \"ee38fb9c-adce-4aae-9896-5b124fac12bf\") " pod="openshift-ingress-canary/ingress-canary-x4jk8"
Jul 28 11:31:22.713030 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.712821    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:31:22.713030 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.712842    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f978d864-c551-4fcf-8a60-ef424a5556a0-config-volume\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:31:22.713030 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.712858    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f978d864-c551-4fcf-8a60-ef424a5556a0-tmp-dir\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:31:22.713030 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.712933    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5bwv\" (UniqueName: \"kubernetes.io/projected/ee38fb9c-adce-4aae-9896-5b124fac12bf-kube-api-access-x5bwv\") pod \"ingress-canary-x4jk8\" (UID: \"ee38fb9c-adce-4aae-9896-5b124fac12bf\") " pod="openshift-ingress-canary/ingress-canary-x4jk8"
Jul 28 11:31:22.814169 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.814098    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlf5q\" (UniqueName: \"kubernetes.io/projected/f978d864-c551-4fcf-8a60-ef424a5556a0-kube-api-access-dlf5q\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:31:22.814169 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.814136    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee38fb9c-adce-4aae-9896-5b124fac12bf-cert\") pod \"ingress-canary-x4jk8\" (UID: \"ee38fb9c-adce-4aae-9896-5b124fac12bf\") " pod="openshift-ingress-canary/ingress-canary-x4jk8"
Jul 28 11:31:22.814169 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.814152    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:31:22.814388 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.814180    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f978d864-c551-4fcf-8a60-ef424a5556a0-config-volume\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:31:22.814388 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.814195    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f978d864-c551-4fcf-8a60-ef424a5556a0-tmp-dir\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:31:22.814388 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:22.814284    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 11:31:22.814388 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:22.814370    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls podName:f978d864-c551-4fcf-8a60-ef424a5556a0 nodeName:}" failed. No retries permitted until 2026-07-28 11:31:23.314345895 +0000 UTC m=+35.173001893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls") pod "dns-default-f7m7q" (UID: "f978d864-c551-4fcf-8a60-ef424a5556a0") : secret "dns-default-metrics-tls" not found
Jul 28 11:31:22.814563 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.814394    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5bwv\" (UniqueName: \"kubernetes.io/projected/ee38fb9c-adce-4aae-9896-5b124fac12bf-kube-api-access-x5bwv\") pod \"ingress-canary-x4jk8\" (UID: \"ee38fb9c-adce-4aae-9896-5b124fac12bf\") " pod="openshift-ingress-canary/ingress-canary-x4jk8"
Jul 28 11:31:22.814563 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.814494    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f978d864-c551-4fcf-8a60-ef424a5556a0-tmp-dir\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:31:22.814805 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.814777    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f978d864-c551-4fcf-8a60-ef424a5556a0-config-volume\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:31:22.818159 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.818141    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee38fb9c-adce-4aae-9896-5b124fac12bf-cert\") pod \"ingress-canary-x4jk8\" (UID: \"ee38fb9c-adce-4aae-9896-5b124fac12bf\") " pod="openshift-ingress-canary/ingress-canary-x4jk8"
Jul 28 11:31:22.832149 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.832120    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5bwv\" (UniqueName: \"kubernetes.io/projected/ee38fb9c-adce-4aae-9896-5b124fac12bf-kube-api-access-x5bwv\") pod \"ingress-canary-x4jk8\" (UID: \"ee38fb9c-adce-4aae-9896-5b124fac12bf\") " pod="openshift-ingress-canary/ingress-canary-x4jk8"
Jul 28 11:31:22.833436 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.833416    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlf5q\" (UniqueName: \"kubernetes.io/projected/f978d864-c551-4fcf-8a60-ef424a5556a0-kube-api-access-dlf5q\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:31:22.882475 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:22.882449    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x4jk8"
Jul 28 11:31:23.257687 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:23.257661    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x4jk8"]
Jul 28 11:31:23.261755 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:31:23.261731    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee38fb9c_adce_4aae_9896_5b124fac12bf.slice/crio-7cde527b1d0fed0ca280b2507b98f013abbfd14f81849de252179deddc889490 WatchSource:0}: Error finding container 7cde527b1d0fed0ca280b2507b98f013abbfd14f81849de252179deddc889490: Status 404 returned error can't find the container with id 7cde527b1d0fed0ca280b2507b98f013abbfd14f81849de252179deddc889490
Jul 28 11:31:23.319255 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:23.319230    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:31:23.319390 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:23.319376    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 11:31:23.319450 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:23.319437    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls podName:f978d864-c551-4fcf-8a60-ef424a5556a0 nodeName:}" failed. No retries permitted until 2026-07-28 11:31:24.31941952 +0000 UTC m=+36.178075499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls") pod "dns-default-f7m7q" (UID: "f978d864-c551-4fcf-8a60-ef424a5556a0") : secret "dns-default-metrics-tls" not found
Jul 28 11:31:23.541349 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:23.541322    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:31:23.544016 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:23.543994    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 11:31:23.709059 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:23.708978    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x4jk8" event={"ID":"ee38fb9c-adce-4aae-9896-5b124fac12bf","Type":"ContainerStarted","Data":"7cde527b1d0fed0ca280b2507b98f013abbfd14f81849de252179deddc889490"}
Jul 28 11:31:23.711903 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:23.711880    2561 generic.go:356] "Generic (PLEG): container finished" podID="bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56" containerID="382c6d95f1984bd022ecce33c475e54a6ae43f45cb9caebc07d8b4ea48aada71" exitCode=0
Jul 28 11:31:23.712001 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:23.711908    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wprpw" event={"ID":"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56","Type":"ContainerDied","Data":"382c6d95f1984bd022ecce33c475e54a6ae43f45cb9caebc07d8b4ea48aada71"}
Jul 28 11:31:24.326327 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:24.326298    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:31:24.326512 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:24.326462    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 11:31:24.326598 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:24.326538    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls podName:f978d864-c551-4fcf-8a60-ef424a5556a0 nodeName:}" failed. No retries permitted until 2026-07-28 11:31:26.326515034 +0000 UTC m=+38.185171012 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls") pod "dns-default-f7m7q" (UID: "f978d864-c551-4fcf-8a60-ef424a5556a0") : secret "dns-default-metrics-tls" not found
Jul 28 11:31:24.718033 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:24.717957    2561 generic.go:356] "Generic (PLEG): container finished" podID="bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56" containerID="c7d274d027def8cd0a81b18b8c3a5fd42fd55511b6b40142a5d34620308e9635" exitCode=0
Jul 28 11:31:24.718033 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:24.718004    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wprpw" event={"ID":"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56","Type":"ContainerDied","Data":"c7d274d027def8cd0a81b18b8c3a5fd42fd55511b6b40142a5d34620308e9635"}
Jul 28 11:31:25.722079 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:25.721831    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x4jk8" event={"ID":"ee38fb9c-adce-4aae-9896-5b124fac12bf","Type":"ContainerStarted","Data":"61be291c2daa76a0515de966f12469b813b193dc1cebdb44b7e6387c845a84e8"}
Jul 28 11:31:25.725158 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:25.725133    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wprpw" event={"ID":"bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56","Type":"ContainerStarted","Data":"e28a3d89eb54f1a04b76fae4063d20f4889e00d7db84c335fe0bb20ab2f57ce2"}
Jul 28 11:31:25.744550 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:25.744506    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x4jk8" podStartSLOduration=1.648343527 podStartE2EDuration="3.744492345s" totalImagesPullingTime="2.096148818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:31:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:31:25.744130064 +0000 UTC m=+37.602786063" watchObservedRunningTime="2026-07-28 11:31:25.744492345 +0000 UTC m=+37.603148379"
Jul 28 11:31:25.780713 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:25.780661    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wprpw" podStartSLOduration=8.618178182 podStartE2EDuration="37.780642858s" totalImagesPullingTime="29.162464676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 11:30:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:31:25.780107853 +0000 UTC m=+37.638763851" watchObservedRunningTime="2026-07-28 11:31:25.780642858 +0000 UTC m=+37.639298881"
Jul 28 11:31:26.035024 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.034952    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx"]
Jul 28 11:31:26.037836 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.037817    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx"
Jul 28 11:31:26.041314 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.041287    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"]
Jul 28 11:31:26.043729 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.043693    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 11:31:26.043896 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.043849    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 11:31:26.044046 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.044019    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-vj2x8\""
Jul 28 11:31:26.044131 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.044051    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 11:31:26.044186 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.044145    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 11:31:26.045036 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.045017    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:26.048784 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:26.048761    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\" is forbidden: User \"system:node:ip-10-0-141-155.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-155.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" type="*v1.Secret"
Jul 28 11:31:26.049156 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:26.049125    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-ca\" is forbidden: User \"system:node:ip-10-0-141-155.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-155.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" type="*v1.Secret"
Jul 28 11:31:26.049247 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:26.049125    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-141-155.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-155.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" type="*v1.Secret"
Jul 28 11:31:26.053895 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.053879    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 11:31:26.062910 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.062890    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx"]
Jul 28 11:31:26.076114 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.076093    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"]
Jul 28 11:31:26.140308 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.140271    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7872f248-f89b-45f1-94fc-70fb87288acf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx\" (UID: \"7872f248-f89b-45f1-94fc-70fb87288acf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx"
Jul 28 11:31:26.140436 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.140310    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg6n7\" (UniqueName: \"kubernetes.io/projected/7872f248-f89b-45f1-94fc-70fb87288acf-kube-api-access-wg6n7\") pod \"managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx\" (UID: \"7872f248-f89b-45f1-94fc-70fb87288acf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx"
Jul 28 11:31:26.240936 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.240906    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0b8d7ed2-882c-41b9-ae46-c30e86605e19-hub\") pod \"cluster-proxy-proxy-agent-8697b44576-v5br9\" (UID: \"0b8d7ed2-882c-41b9-ae46-c30e86605e19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:26.241042 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.240965    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b8d7ed2-882c-41b9-ae46-c30e86605e19-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8697b44576-v5br9\" (UID: \"0b8d7ed2-882c-41b9-ae46-c30e86605e19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:26.241042 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.241020    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7872f248-f89b-45f1-94fc-70fb87288acf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx\" (UID: \"7872f248-f89b-45f1-94fc-70fb87288acf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx"
Jul 28 11:31:26.241123 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.241046    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg6n7\" (UniqueName: \"kubernetes.io/projected/7872f248-f89b-45f1-94fc-70fb87288acf-kube-api-access-wg6n7\") pod \"managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx\" (UID: \"7872f248-f89b-45f1-94fc-70fb87288acf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx"
Jul 28 11:31:26.241123 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.241068    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b8d7ed2-882c-41b9-ae46-c30e86605e19-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8697b44576-v5br9\" (UID: \"0b8d7ed2-882c-41b9-ae46-c30e86605e19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:26.241123 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.241092    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq7sp\" (UniqueName: \"kubernetes.io/projected/0b8d7ed2-882c-41b9-ae46-c30e86605e19-kube-api-access-kq7sp\") pod \"cluster-proxy-proxy-agent-8697b44576-v5br9\" (UID: \"0b8d7ed2-882c-41b9-ae46-c30e86605e19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:26.241123 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.241120    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0b8d7ed2-882c-41b9-ae46-c30e86605e19-ca\") pod \"cluster-proxy-proxy-agent-8697b44576-v5br9\" (UID: \"0b8d7ed2-882c-41b9-ae46-c30e86605e19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:26.241242 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.241137    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0b8d7ed2-882c-41b9-ae46-c30e86605e19-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8697b44576-v5br9\" (UID: \"0b8d7ed2-882c-41b9-ae46-c30e86605e19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:26.245277 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.245257    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7872f248-f89b-45f1-94fc-70fb87288acf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx\" (UID: \"7872f248-f89b-45f1-94fc-70fb87288acf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx"
Jul 28 11:31:26.255071 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.255050    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg6n7\" (UniqueName: \"kubernetes.io/projected/7872f248-f89b-45f1-94fc-70fb87288acf-kube-api-access-wg6n7\") pod \"managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx\" (UID: \"7872f248-f89b-45f1-94fc-70fb87288acf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx"
Jul 28 11:31:26.342065 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.342029    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b8d7ed2-882c-41b9-ae46-c30e86605e19-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8697b44576-v5br9\" (UID: \"0b8d7ed2-882c-41b9-ae46-c30e86605e19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:26.342065 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.342065    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq7sp\" (UniqueName: \"kubernetes.io/projected/0b8d7ed2-882c-41b9-ae46-c30e86605e19-kube-api-access-kq7sp\") pod \"cluster-proxy-proxy-agent-8697b44576-v5br9\" (UID: \"0b8d7ed2-882c-41b9-ae46-c30e86605e19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:26.342254 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.342085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0b8d7ed2-882c-41b9-ae46-c30e86605e19-ca\") pod \"cluster-proxy-proxy-agent-8697b44576-v5br9\" (UID: \"0b8d7ed2-882c-41b9-ae46-c30e86605e19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:26.342254 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.342101    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0b8d7ed2-882c-41b9-ae46-c30e86605e19-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8697b44576-v5br9\" (UID: \"0b8d7ed2-882c-41b9-ae46-c30e86605e19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:26.342254 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.342126    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:31:26.342254 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.342144    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0b8d7ed2-882c-41b9-ae46-c30e86605e19-hub\") pod \"cluster-proxy-proxy-agent-8697b44576-v5br9\" (UID: \"0b8d7ed2-882c-41b9-ae46-c30e86605e19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:26.342254 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:26.342227    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 11:31:26.342482 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.342278    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b8d7ed2-882c-41b9-ae46-c30e86605e19-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8697b44576-v5br9\" (UID: \"0b8d7ed2-882c-41b9-ae46-c30e86605e19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:26.342482 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:26.342290    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls podName:f978d864-c551-4fcf-8a60-ef424a5556a0 nodeName:}" failed. No retries permitted until 2026-07-28 11:31:30.342275431 +0000 UTC m=+42.200931408 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls") pod "dns-default-f7m7q" (UID: "f978d864-c551-4fcf-8a60-ef424a5556a0") : secret "dns-default-metrics-tls" not found
Jul 28 11:31:26.342709 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.342692    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b8d7ed2-882c-41b9-ae46-c30e86605e19-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8697b44576-v5br9\" (UID: \"0b8d7ed2-882c-41b9-ae46-c30e86605e19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:26.344342 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.344322    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0b8d7ed2-882c-41b9-ae46-c30e86605e19-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8697b44576-v5br9\" (UID: \"0b8d7ed2-882c-41b9-ae46-c30e86605e19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:26.351406 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.351385    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq7sp\" (UniqueName: \"kubernetes.io/projected/0b8d7ed2-882c-41b9-ae46-c30e86605e19-kube-api-access-kq7sp\") pod \"cluster-proxy-proxy-agent-8697b44576-v5br9\" (UID: \"0b8d7ed2-882c-41b9-ae46-c30e86605e19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:26.354602 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.354567    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx"
Jul 28 11:31:26.479742 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.479719    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx"]
Jul 28 11:31:26.482791 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:31:26.482763    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7872f248_f89b_45f1_94fc_70fb87288acf.slice/crio-207a0d02ee5065af0489a1fff9d6ae65534ddbbcf92c2a292aebf8b89672c496 WatchSource:0}: Error finding container 207a0d02ee5065af0489a1fff9d6ae65534ddbbcf92c2a292aebf8b89672c496: Status 404 returned error can't find the container with id 207a0d02ee5065af0489a1fff9d6ae65534ddbbcf92c2a292aebf8b89672c496
Jul 28 11:31:26.728168 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:26.728086    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx" event={"ID":"7872f248-f89b-45f1-94fc-70fb87288acf","Type":"ContainerStarted","Data":"207a0d02ee5065af0489a1fff9d6ae65534ddbbcf92c2a292aebf8b89672c496"}
Jul 28 11:31:27.173231 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:27.173200    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 11:31:27.185469 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:27.185444    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b8d7ed2-882c-41b9-ae46-c30e86605e19-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8697b44576-v5br9\" (UID: \"0b8d7ed2-882c-41b9-ae46-c30e86605e19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:27.342738 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:27.342705    2561 secret.go:189] Couldn't get secret open-cluster-management-agent-addon/cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert: failed to sync secret cache: timed out waiting for the condition
Jul 28 11:31:27.342895 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:27.342715    2561 secret.go:189] Couldn't get secret open-cluster-management-agent-addon/cluster-proxy-ca: failed to sync secret cache: timed out waiting for the condition
Jul 28 11:31:27.342895 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:27.342819    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b8d7ed2-882c-41b9-ae46-c30e86605e19-hub podName:0b8d7ed2-882c-41b9-ae46-c30e86605e19 nodeName:}" failed. No retries permitted until 2026-07-28 11:31:27.842794398 +0000 UTC m=+39.701450379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "hub" (UniqueName: "kubernetes.io/secret/0b8d7ed2-882c-41b9-ae46-c30e86605e19-hub") pod "cluster-proxy-proxy-agent-8697b44576-v5br9" (UID: "0b8d7ed2-882c-41b9-ae46-c30e86605e19") : failed to sync secret cache: timed out waiting for the condition
Jul 28 11:31:27.343030 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:27.342909    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b8d7ed2-882c-41b9-ae46-c30e86605e19-ca podName:0b8d7ed2-882c-41b9-ae46-c30e86605e19 nodeName:}" failed. No retries permitted until 2026-07-28 11:31:27.842887844 +0000 UTC m=+39.701543835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ca" (UniqueName: "kubernetes.io/secret/0b8d7ed2-882c-41b9-ae46-c30e86605e19-ca") pod "cluster-proxy-proxy-agent-8697b44576-v5br9" (UID: "0b8d7ed2-882c-41b9-ae46-c30e86605e19") : failed to sync secret cache: timed out waiting for the condition
Jul 28 11:31:27.533691 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:27.533573    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 11:31:27.583867 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:27.583837    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 11:31:27.855225 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:27.855192    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0b8d7ed2-882c-41b9-ae46-c30e86605e19-hub\") pod \"cluster-proxy-proxy-agent-8697b44576-v5br9\" (UID: \"0b8d7ed2-882c-41b9-ae46-c30e86605e19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:27.855682 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:27.855285    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0b8d7ed2-882c-41b9-ae46-c30e86605e19-ca\") pod \"cluster-proxy-proxy-agent-8697b44576-v5br9\" (UID: \"0b8d7ed2-882c-41b9-ae46-c30e86605e19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:27.858431 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:27.858386    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0b8d7ed2-882c-41b9-ae46-c30e86605e19-ca\") pod \"cluster-proxy-proxy-agent-8697b44576-v5br9\" (UID: \"0b8d7ed2-882c-41b9-ae46-c30e86605e19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:27.858594 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:27.858438    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0b8d7ed2-882c-41b9-ae46-c30e86605e19-hub\") pod \"cluster-proxy-proxy-agent-8697b44576-v5br9\" (UID: \"0b8d7ed2-882c-41b9-ae46-c30e86605e19\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:27.861262 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:27.861239    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:31:28.026135 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:28.026069    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"]
Jul 28 11:31:29.027079 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:31:29.027023    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b8d7ed2_882c_41b9_ae46_c30e86605e19.slice/crio-e197180117b9263d069857f0104bed6e4ec929e972ac37d952892e17acf13927 WatchSource:0}: Error finding container e197180117b9263d069857f0104bed6e4ec929e972ac37d952892e17acf13927: Status 404 returned error can't find the container with id e197180117b9263d069857f0104bed6e4ec929e972ac37d952892e17acf13927
Jul 28 11:31:29.737811 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:29.737775    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx" event={"ID":"7872f248-f89b-45f1-94fc-70fb87288acf","Type":"ContainerStarted","Data":"319b5685816e513a71063818f25623c4b23af7fdb90721ca3edb7329184d1649"}
Jul 28 11:31:29.738850 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:29.738827    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9" event={"ID":"0b8d7ed2-882c-41b9-ae46-c30e86605e19","Type":"ContainerStarted","Data":"e197180117b9263d069857f0104bed6e4ec929e972ac37d952892e17acf13927"}
Jul 28 11:31:29.756543 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:29.756503    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx" podStartSLOduration=1.151104136 podStartE2EDuration="3.756489357s" totalImagesPullingTime="2.605385221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:31:29.756057215 +0000 UTC m=+41.614713215" watchObservedRunningTime="2026-07-28 11:31:29.756489357 +0000 UTC m=+41.615145355"
Jul 28 11:31:30.372867 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:30.372825    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:31:30.373267 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:30.372987    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 11:31:30.373267 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:30.373071    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls podName:f978d864-c551-4fcf-8a60-ef424a5556a0 nodeName:}" failed. No retries permitted until 2026-07-28 11:31:38.373049208 +0000 UTC m=+50.231705188 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls") pod "dns-default-f7m7q" (UID: "f978d864-c551-4fcf-8a60-ef424a5556a0") : secret "dns-default-metrics-tls" not found
Jul 28 11:31:33.750753 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:33.750670    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9" event={"ID":"0b8d7ed2-882c-41b9-ae46-c30e86605e19","Type":"ContainerStarted","Data":"495201c25a55fe7b74141368e8cb954cfa8f4c28ae07b687d9c79161fdcaac43"}
Jul 28 11:31:33.750753 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:33.750720    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9" event={"ID":"0b8d7ed2-882c-41b9-ae46-c30e86605e19","Type":"ContainerStarted","Data":"a016f99f4a0df61d4e971c3a0a597f4eaf400545e630f374ce29964ff02c5cfb"}
Jul 28 11:31:33.750753 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:33.750740    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9" event={"ID":"0b8d7ed2-882c-41b9-ae46-c30e86605e19","Type":"ContainerStarted","Data":"dcd3dabb2cf9045a6a0aa87362030b42442a48811334c9d7cf040a30ce8c5019"}
Jul 28 11:31:38.430590 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:38.430551    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:31:38.430952 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:38.430701    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 11:31:38.430952 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:38.430771    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls podName:f978d864-c551-4fcf-8a60-ef424a5556a0 nodeName:}" failed. No retries permitted until 2026-07-28 11:31:54.430754201 +0000 UTC m=+66.289410179 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls") pod "dns-default-f7m7q" (UID: "f978d864-c551-4fcf-8a60-ef424a5556a0") : secret "dns-default-metrics-tls" not found
Jul 28 11:31:47.700781 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:47.700754    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f2c2r"
Jul 28 11:31:47.736165 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:47.736118    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9" podStartSLOduration=17.682326465 podStartE2EDuration="21.736103602s" totalImagesPullingTime="4.053777137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:31:33.779316872 +0000 UTC m=+45.637972891" watchObservedRunningTime="2026-07-28 11:31:47.736103602 +0000 UTC m=+59.594759578"
Jul 28 11:31:54.231817 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:54.231767    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs\") pod \"network-metrics-daemon-ss7h8\" (UID: \"85d71f38-a39a-4a1c-90dc-f5b941567052\") " pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:31:54.234312 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:54.234295    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 11:31:54.242166 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:54.242148    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 11:31:54.242241 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:54.242213    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs podName:85d71f38-a39a-4a1c-90dc-f5b941567052 nodeName:}" failed. No retries permitted until 2026-07-28 11:32:58.242192267 +0000 UTC m=+130.100848244 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs") pod "network-metrics-daemon-ss7h8" (UID: "85d71f38-a39a-4a1c-90dc-f5b941567052") : secret "metrics-daemon-secret" not found
Jul 28 11:31:54.332681 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:54.332650    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret\") pod \"global-pull-secret-syncer-fg8rp\" (UID: \"1a04e1e5-57de-4b91-b28e-f4ef1b5e67be\") " pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:31:54.332792 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:54.332718    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88qq\" (UniqueName: \"kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq\") pod \"network-check-target-bcd9g\" (UID: \"35b9e83d-edd8-4afa-9368-a7a490338b85\") " pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:31:54.335779 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:54.335758    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 11:31:54.335864 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:54.335784    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 11:31:54.345871 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:54.345847    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a04e1e5-57de-4b91-b28e-f4ef1b5e67be-original-pull-secret\") pod \"global-pull-secret-syncer-fg8rp\" (UID: \"1a04e1e5-57de-4b91-b28e-f4ef1b5e67be\") " pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:31:54.346143 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:54.346118    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 11:31:54.356181 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:54.356160    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d88qq\" (UniqueName: \"kubernetes.io/projected/35b9e83d-edd8-4afa-9368-a7a490338b85-kube-api-access-d88qq\") pod \"network-check-target-bcd9g\" (UID: \"35b9e83d-edd8-4afa-9368-a7a490338b85\") " pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:31:54.397014 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:54.396994    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tk4rs\""
Jul 28 11:31:54.404895 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:54.404881    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:31:54.433286 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:54.433257    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:31:54.433416 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:54.433399    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 11:31:54.433482 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:31:54.433472    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls podName:f978d864-c551-4fcf-8a60-ef424a5556a0 nodeName:}" failed. No retries permitted until 2026-07-28 11:32:26.433455114 +0000 UTC m=+98.292111091 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls") pod "dns-default-f7m7q" (UID: "f978d864-c551-4fcf-8a60-ef424a5556a0") : secret "dns-default-metrics-tls" not found
Jul 28 11:31:54.449933 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:54.449856    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg8rp"
Jul 28 11:31:54.531971 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:54.531942    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bcd9g"]
Jul 28 11:31:54.536462 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:31:54.536439    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b9e83d_edd8_4afa_9368_a7a490338b85.slice/crio-1cc452623b266f4897cd94985e4077225dd7910e3600554ca93a5df0dc775e68 WatchSource:0}: Error finding container 1cc452623b266f4897cd94985e4077225dd7910e3600554ca93a5df0dc775e68: Status 404 returned error can't find the container with id 1cc452623b266f4897cd94985e4077225dd7910e3600554ca93a5df0dc775e68
Jul 28 11:31:54.578641 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:54.578616    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fg8rp"]
Jul 28 11:31:54.581513 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:31:54.581489    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a04e1e5_57de_4b91_b28e_f4ef1b5e67be.slice/crio-e609b12b3fc9e259f7344eadc31f5f71c406bda06114de25918868a655544497 WatchSource:0}: Error finding container e609b12b3fc9e259f7344eadc31f5f71c406bda06114de25918868a655544497: Status 404 returned error can't find the container with id e609b12b3fc9e259f7344eadc31f5f71c406bda06114de25918868a655544497
Jul 28 11:31:54.804064 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:54.804030    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fg8rp" event={"ID":"1a04e1e5-57de-4b91-b28e-f4ef1b5e67be","Type":"ContainerStarted","Data":"e609b12b3fc9e259f7344eadc31f5f71c406bda06114de25918868a655544497"}
Jul 28 11:31:54.804959 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:54.804937    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bcd9g" event={"ID":"35b9e83d-edd8-4afa-9368-a7a490338b85","Type":"ContainerStarted","Data":"1cc452623b266f4897cd94985e4077225dd7910e3600554ca93a5df0dc775e68"}
Jul 28 11:31:59.821894 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:59.821853    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bcd9g" event={"ID":"35b9e83d-edd8-4afa-9368-a7a490338b85","Type":"ContainerStarted","Data":"c013e752fc6b38291799efeed1b9afb78fd1222fda99ab727083953582eeb3ab"}
Jul 28 11:31:59.822379 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:59.821972    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:31:59.823232 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:59.823210    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fg8rp" event={"ID":"1a04e1e5-57de-4b91-b28e-f4ef1b5e67be","Type":"ContainerStarted","Data":"f8979cc2455eaa330c695062e9f94cd04482acdc1561d0e51c0706e8449bb674"}
Jul 28 11:31:59.837089 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:59.837048    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bcd9g" podStartSLOduration=67.006257657 podStartE2EDuration="1m11.837035104s" totalImagesPullingTime="4.830777447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:31:59.836626114 +0000 UTC m=+71.695282114" watchObservedRunningTime="2026-07-28 11:31:59.837035104 +0000 UTC m=+71.695691103"
Jul 28 11:31:59.850772 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:31:59.850731    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fg8rp" podStartSLOduration=67.053609185 podStartE2EDuration="1m11.85070282s" totalImagesPullingTime="4.797093635s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 11:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:31:59.850154087 +0000 UTC m=+71.708810087" watchObservedRunningTime="2026-07-28 11:31:59.85070282 +0000 UTC m=+71.709358821"
Jul 28 11:32:26.443842 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:32:26.443800    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:32:26.444310 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:32:26.443915    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 11:32:26.444310 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:32:26.443969    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls podName:f978d864-c551-4fcf-8a60-ef424a5556a0 nodeName:}" failed. No retries permitted until 2026-07-28 11:33:30.443955118 +0000 UTC m=+162.302611095 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls") pod "dns-default-f7m7q" (UID: "f978d864-c551-4fcf-8a60-ef424a5556a0") : secret "dns-default-metrics-tls" not found
Jul 28 11:32:30.828324 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:32:30.828287    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bcd9g"
Jul 28 11:32:58.259389 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:32:58.259334    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs\") pod \"network-metrics-daemon-ss7h8\" (UID: \"85d71f38-a39a-4a1c-90dc-f5b941567052\") " pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:32:58.259892 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:32:58.259480    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 11:32:58.259892 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:32:58.259547    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs podName:85d71f38-a39a-4a1c-90dc-f5b941567052 nodeName:}" failed. No retries permitted until 2026-07-28 11:35:00.25953083 +0000 UTC m=+252.118186809 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs") pod "network-metrics-daemon-ss7h8" (UID: "85d71f38-a39a-4a1c-90dc-f5b941567052") : secret "metrics-daemon-secret" not found
Jul 28 11:33:25.566972 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:33:25.566931    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-f7m7q" podUID="f978d864-c551-4fcf-8a60-ef424a5556a0"
Jul 28 11:33:25.589260 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:33:25.589232    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ss7h8" podUID="85d71f38-a39a-4a1c-90dc-f5b941567052"
Jul 28 11:33:26.061361 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:33:26.061331    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:33:30.073800 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:33:30.073769    2561 generic.go:356] "Generic (PLEG): container finished" podID="7872f248-f89b-45f1-94fc-70fb87288acf" containerID="319b5685816e513a71063818f25623c4b23af7fdb90721ca3edb7329184d1649" exitCode=255
Jul 28 11:33:30.074212 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:33:30.073815    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx" event={"ID":"7872f248-f89b-45f1-94fc-70fb87288acf","Type":"ContainerDied","Data":"319b5685816e513a71063818f25623c4b23af7fdb90721ca3edb7329184d1649"}
Jul 28 11:33:30.074212 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:33:30.074100    2561 scope.go:117] "RemoveContainer" containerID="319b5685816e513a71063818f25623c4b23af7fdb90721ca3edb7329184d1649"
Jul 28 11:33:30.469644 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:33:30.469554    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:33:30.469777 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:33:30.469709    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 11:33:30.469777 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:33:30.469772    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls podName:f978d864-c551-4fcf-8a60-ef424a5556a0 nodeName:}" failed. No retries permitted until 2026-07-28 11:35:32.469756387 +0000 UTC m=+284.328412364 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls") pod "dns-default-f7m7q" (UID: "f978d864-c551-4fcf-8a60-ef424a5556a0") : secret "dns-default-metrics-tls" not found
Jul 28 11:33:31.078931 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:33:31.078895    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx" event={"ID":"7872f248-f89b-45f1-94fc-70fb87288acf","Type":"ContainerStarted","Data":"5d8738ea12755868ece1ec797d12a9caaf1ceb60d19225d1435047b28a2a0cc1"}
Jul 28 11:33:39.540398 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:33:39.540365    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:34:57.882870 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:57.882834    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-88547"]
Jul 28 11:34:57.885777 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:57.885761    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:34:57.888119 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:57.888097    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 11:34:57.888231 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:57.888151    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 11:34:57.888231 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:57.888196    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 11:34:57.888432 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:57.888414    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 11:34:57.889152 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:57.889138    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l2vtn\""
Jul 28 11:34:57.899811 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:57.899788    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-88547"]
Jul 28 11:34:58.043498 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:58.043461    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/34712309-48b9-4e19-a7dc-92be37026a55-data-volume\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:34:58.043678 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:58.043517    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:34:58.043678 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:58.043630    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34712309-48b9-4e19-a7dc-92be37026a55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:34:58.043760 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:58.043684    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9ccs\" (UniqueName: \"kubernetes.io/projected/34712309-48b9-4e19-a7dc-92be37026a55-kube-api-access-j9ccs\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:34:58.043760 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:58.043712    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/34712309-48b9-4e19-a7dc-92be37026a55-crio-socket\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:34:58.144428 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:58.144357    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34712309-48b9-4e19-a7dc-92be37026a55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:34:58.144428 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:58.144404    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9ccs\" (UniqueName: \"kubernetes.io/projected/34712309-48b9-4e19-a7dc-92be37026a55-kube-api-access-j9ccs\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:34:58.144613 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:58.144434    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/34712309-48b9-4e19-a7dc-92be37026a55-crio-socket\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:34:58.144613 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:34:58.144468    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 11:34:58.144613 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:58.144469    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/34712309-48b9-4e19-a7dc-92be37026a55-data-volume\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:34:58.144613 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:34:58.144535    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34712309-48b9-4e19-a7dc-92be37026a55-kube-rbac-proxy-cm podName:34712309-48b9-4e19-a7dc-92be37026a55 nodeName:}" failed. No retries permitted until 2026-07-28 11:34:58.644517403 +0000 UTC m=+250.503173384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/34712309-48b9-4e19-a7dc-92be37026a55-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88547" (UID: "34712309-48b9-4e19-a7dc-92be37026a55") : configmap "kube-rbac-proxy" not found
Jul 28 11:34:58.144613 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:58.144575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:34:58.144613 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:58.144598    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/34712309-48b9-4e19-a7dc-92be37026a55-crio-socket\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:34:58.144898 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:34:58.144692    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 11:34:58.144898 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:34:58.144727    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls podName:34712309-48b9-4e19-a7dc-92be37026a55 nodeName:}" failed. No retries permitted until 2026-07-28 11:34:58.644717982 +0000 UTC m=+250.503373959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88547" (UID: "34712309-48b9-4e19-a7dc-92be37026a55") : secret "insights-runtime-extractor-tls" not found
Jul 28 11:34:58.144898 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:58.144815    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/34712309-48b9-4e19-a7dc-92be37026a55-data-volume\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:34:58.157605 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:58.157573    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9ccs\" (UniqueName: \"kubernetes.io/projected/34712309-48b9-4e19-a7dc-92be37026a55-kube-api-access-j9ccs\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:34:58.646962 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:58.646921    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:34:58.647150 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:58.646979    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34712309-48b9-4e19-a7dc-92be37026a55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:34:58.647150 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:34:58.647047    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 11:34:58.647150 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:34:58.647087    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 11:34:58.647150 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:34:58.647113    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34712309-48b9-4e19-a7dc-92be37026a55-kube-rbac-proxy-cm podName:34712309-48b9-4e19-a7dc-92be37026a55 nodeName:}" failed. No retries permitted until 2026-07-28 11:34:59.64710016 +0000 UTC m=+251.505756136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/34712309-48b9-4e19-a7dc-92be37026a55-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88547" (UID: "34712309-48b9-4e19-a7dc-92be37026a55") : configmap "kube-rbac-proxy" not found
Jul 28 11:34:58.647309 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:34:58.647156    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls podName:34712309-48b9-4e19-a7dc-92be37026a55 nodeName:}" failed. No retries permitted until 2026-07-28 11:34:59.647139351 +0000 UTC m=+251.505795329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88547" (UID: "34712309-48b9-4e19-a7dc-92be37026a55") : secret "insights-runtime-extractor-tls" not found
Jul 28 11:34:59.654590 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:59.654552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34712309-48b9-4e19-a7dc-92be37026a55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:34:59.655046 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:34:59.654630    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:34:59.655046 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:34:59.654676    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 11:34:59.655046 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:34:59.654757    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34712309-48b9-4e19-a7dc-92be37026a55-kube-rbac-proxy-cm podName:34712309-48b9-4e19-a7dc-92be37026a55 nodeName:}" failed. No retries permitted until 2026-07-28 11:35:01.654736366 +0000 UTC m=+253.513392364 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/34712309-48b9-4e19-a7dc-92be37026a55-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88547" (UID: "34712309-48b9-4e19-a7dc-92be37026a55") : configmap "kube-rbac-proxy" not found
Jul 28 11:34:59.655046 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:34:59.654759    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 11:34:59.655046 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:34:59.654806    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls podName:34712309-48b9-4e19-a7dc-92be37026a55 nodeName:}" failed. No retries permitted until 2026-07-28 11:35:01.654793803 +0000 UTC m=+253.513449781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88547" (UID: "34712309-48b9-4e19-a7dc-92be37026a55") : secret "insights-runtime-extractor-tls" not found
Jul 28 11:35:00.360228 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:00.360196    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs\") pod \"network-metrics-daemon-ss7h8\" (UID: \"85d71f38-a39a-4a1c-90dc-f5b941567052\") " pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:35:00.360392 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:35:00.360339    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 11:35:00.360442 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:35:00.360406    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs podName:85d71f38-a39a-4a1c-90dc-f5b941567052 nodeName:}" failed. No retries permitted until 2026-07-28 11:37:02.360391064 +0000 UTC m=+374.219047041 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs") pod "network-metrics-daemon-ss7h8" (UID: "85d71f38-a39a-4a1c-90dc-f5b941567052") : secret "metrics-daemon-secret" not found
Jul 28 11:35:00.664403 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:00.664330    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hpn2d_e552bdf3-9456-4e29-b6c8-aaad902d9d5c/dns-node-resolver/0.log"
Jul 28 11:35:01.669207 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:01.669176    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34712309-48b9-4e19-a7dc-92be37026a55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:35:01.669625 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:01.669234    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:35:01.669625 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:35:01.669299    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 11:35:01.669625 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:35:01.669329    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 11:35:01.669625 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:35:01.669367    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34712309-48b9-4e19-a7dc-92be37026a55-kube-rbac-proxy-cm podName:34712309-48b9-4e19-a7dc-92be37026a55 nodeName:}" failed. No retries permitted until 2026-07-28 11:35:05.669347704 +0000 UTC m=+257.528003697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/34712309-48b9-4e19-a7dc-92be37026a55-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88547" (UID: "34712309-48b9-4e19-a7dc-92be37026a55") : configmap "kube-rbac-proxy" not found
Jul 28 11:35:01.669625 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:35:01.669380    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls podName:34712309-48b9-4e19-a7dc-92be37026a55 nodeName:}" failed. No retries permitted until 2026-07-28 11:35:05.66937481 +0000 UTC m=+257.528030791 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88547" (UID: "34712309-48b9-4e19-a7dc-92be37026a55") : secret "insights-runtime-extractor-tls" not found
Jul 28 11:35:01.670351 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:01.670333    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pthzf_9af318d1-7622-4574-8c01-ee23cbcf4fdc/node-ca/0.log"
Jul 28 11:35:02.665121 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:02.665094    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x4jk8_ee38fb9c-adce-4aae-9896-5b124fac12bf/serve-healthcheck-canary/0.log"
Jul 28 11:35:05.698277 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:05.698237    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34712309-48b9-4e19-a7dc-92be37026a55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:35:05.698738 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:05.698343    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:35:05.698738 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:35:05.698383    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 11:35:05.698738 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:35:05.698466    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/34712309-48b9-4e19-a7dc-92be37026a55-kube-rbac-proxy-cm podName:34712309-48b9-4e19-a7dc-92be37026a55 nodeName:}" failed. No retries permitted until 2026-07-28 11:35:13.698445619 +0000 UTC m=+265.557101601 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/34712309-48b9-4e19-a7dc-92be37026a55-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88547" (UID: "34712309-48b9-4e19-a7dc-92be37026a55") : configmap "kube-rbac-proxy" not found
Jul 28 11:35:05.698738 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:35:05.698473    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 11:35:05.698738 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:35:05.698523    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls podName:34712309-48b9-4e19-a7dc-92be37026a55 nodeName:}" failed. No retries permitted until 2026-07-28 11:35:13.698508145 +0000 UTC m=+265.557164125 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88547" (UID: "34712309-48b9-4e19-a7dc-92be37026a55") : secret "insights-runtime-extractor-tls" not found
Jul 28 11:35:13.763031 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:13.762947    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:35:13.763031 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:13.762997    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34712309-48b9-4e19-a7dc-92be37026a55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:35:13.763525 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:35:13.763115    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 11:35:13.763525 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:35:13.763211    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls podName:34712309-48b9-4e19-a7dc-92be37026a55 nodeName:}" failed. No retries permitted until 2026-07-28 11:35:29.763190051 +0000 UTC m=+281.621846043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88547" (UID: "34712309-48b9-4e19-a7dc-92be37026a55") : secret "insights-runtime-extractor-tls" not found
Jul 28 11:35:13.763525 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:13.763471    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34712309-48b9-4e19-a7dc-92be37026a55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:35:22.602100 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.602064    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c46895c78-mnn2v"]
Jul 28 11:35:22.606778 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.606752    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.609253 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.609226    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 11:35:22.609418 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.609226    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 11:35:22.609486 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.609458    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cxmgd\""
Jul 28 11:35:22.609610 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.609573    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 11:35:22.615353 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.615335    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 11:35:22.620679 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.620656    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c46895c78-mnn2v"]
Jul 28 11:35:22.722926 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.722896    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b4b1d25-a678-4bfa-8320-d0210806a661-bound-sa-token\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.722926 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.722932    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b4b1d25-a678-4bfa-8320-d0210806a661-registry-tls\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.723120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.722952    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0b4b1d25-a678-4bfa-8320-d0210806a661-ca-trust-extracted\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.723120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.723019    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0b4b1d25-a678-4bfa-8320-d0210806a661-installation-pull-secrets\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.723120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.723053    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0b4b1d25-a678-4bfa-8320-d0210806a661-registry-certificates\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.723120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.723081    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftlg7\" (UniqueName: \"kubernetes.io/projected/0b4b1d25-a678-4bfa-8320-d0210806a661-kube-api-access-ftlg7\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.723120 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.723102    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0b4b1d25-a678-4bfa-8320-d0210806a661-image-registry-private-configuration\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.723270 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.723164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b4b1d25-a678-4bfa-8320-d0210806a661-trusted-ca\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.823537 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.823509    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0b4b1d25-a678-4bfa-8320-d0210806a661-registry-certificates\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.823537 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.823540    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftlg7\" (UniqueName: \"kubernetes.io/projected/0b4b1d25-a678-4bfa-8320-d0210806a661-kube-api-access-ftlg7\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.823762 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.823559    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0b4b1d25-a678-4bfa-8320-d0210806a661-image-registry-private-configuration\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.823762 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.823648    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b4b1d25-a678-4bfa-8320-d0210806a661-trusted-ca\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.823762 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.823707    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b4b1d25-a678-4bfa-8320-d0210806a661-bound-sa-token\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.823762 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.823734    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b4b1d25-a678-4bfa-8320-d0210806a661-registry-tls\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.823962 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.823764    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0b4b1d25-a678-4bfa-8320-d0210806a661-ca-trust-extracted\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.823962 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.823810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0b4b1d25-a678-4bfa-8320-d0210806a661-installation-pull-secrets\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.824318 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.824285    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0b4b1d25-a678-4bfa-8320-d0210806a661-ca-trust-extracted\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.824657 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.824639    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0b4b1d25-a678-4bfa-8320-d0210806a661-registry-certificates\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.824882 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.824861    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b4b1d25-a678-4bfa-8320-d0210806a661-trusted-ca\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.826152 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.826128    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0b4b1d25-a678-4bfa-8320-d0210806a661-installation-pull-secrets\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.826240 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.826152    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0b4b1d25-a678-4bfa-8320-d0210806a661-image-registry-private-configuration\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.826313 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.826297    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b4b1d25-a678-4bfa-8320-d0210806a661-registry-tls\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.834732 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.834704    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftlg7\" (UniqueName: \"kubernetes.io/projected/0b4b1d25-a678-4bfa-8320-d0210806a661-kube-api-access-ftlg7\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.835769 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.835752    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b4b1d25-a678-4bfa-8320-d0210806a661-bound-sa-token\") pod \"image-registry-5c46895c78-mnn2v\" (UID: \"0b4b1d25-a678-4bfa-8320-d0210806a661\") " pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:22.916010 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:22.915955    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:23.039892 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:23.039865    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c46895c78-mnn2v"]
Jul 28 11:35:23.042312 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:35:23.042288    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4b1d25_a678_4bfa_8320_d0210806a661.slice/crio-69c4343fb3cecda4baa833e22598bc67a2a5439b23805c9a9c8d81c2fbb58441 WatchSource:0}: Error finding container 69c4343fb3cecda4baa833e22598bc67a2a5439b23805c9a9c8d81c2fbb58441: Status 404 returned error can't find the container with id 69c4343fb3cecda4baa833e22598bc67a2a5439b23805c9a9c8d81c2fbb58441
Jul 28 11:35:23.388522 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:23.388487    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c46895c78-mnn2v" event={"ID":"0b4b1d25-a678-4bfa-8320-d0210806a661","Type":"ContainerStarted","Data":"f24dd67a625051f7b4fc1f1d957dda7d21dd2cc36db34ab052dabe1b400cdc5c"}
Jul 28 11:35:23.388702 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:23.388538    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c46895c78-mnn2v" event={"ID":"0b4b1d25-a678-4bfa-8320-d0210806a661","Type":"ContainerStarted","Data":"69c4343fb3cecda4baa833e22598bc67a2a5439b23805c9a9c8d81c2fbb58441"}
Jul 28 11:35:23.388702 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:23.388566    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:23.408339 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:23.408293    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c46895c78-mnn2v" podStartSLOduration=1.40827518 podStartE2EDuration="1.40827518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:35:23.40755901 +0000 UTC m=+275.266215020" watchObservedRunningTime="2026-07-28 11:35:23.40827518 +0000 UTC m=+275.266931181"
Jul 28 11:35:27.862814 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:27.862769    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9" podUID="0b8d7ed2-882c-41b9-ae46-c30e86605e19" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 11:35:29.062528 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:35:29.062475    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-f7m7q" podUID="f978d864-c551-4fcf-8a60-ef424a5556a0"
Jul 28 11:35:29.404772 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:29.404697    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:35:29.774416 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:29.774325    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:35:29.776486 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:29.776454    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34712309-48b9-4e19-a7dc-92be37026a55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88547\" (UID: \"34712309-48b9-4e19-a7dc-92be37026a55\") " pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:35:29.994473 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:29.994429    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-88547"
Jul 28 11:35:30.121290 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:30.121257    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-88547"]
Jul 28 11:35:30.124466 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:35:30.124437    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34712309_48b9_4e19_a7dc_92be37026a55.slice/crio-1fbcb9656f7705f880d666675418815ce2631faf8fab93233ba81379566ae0cf WatchSource:0}: Error finding container 1fbcb9656f7705f880d666675418815ce2631faf8fab93233ba81379566ae0cf: Status 404 returned error can't find the container with id 1fbcb9656f7705f880d666675418815ce2631faf8fab93233ba81379566ae0cf
Jul 28 11:35:30.409401 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:30.409367    2561 generic.go:356] "Generic (PLEG): container finished" podID="7872f248-f89b-45f1-94fc-70fb87288acf" containerID="5d8738ea12755868ece1ec797d12a9caaf1ceb60d19225d1435047b28a2a0cc1" exitCode=255
Jul 28 11:35:30.409602 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:30.409442    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx" event={"ID":"7872f248-f89b-45f1-94fc-70fb87288acf","Type":"ContainerDied","Data":"5d8738ea12755868ece1ec797d12a9caaf1ceb60d19225d1435047b28a2a0cc1"}
Jul 28 11:35:30.410291 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:30.410271    2561 scope.go:117] "RemoveContainer" containerID="319b5685816e513a71063818f25623c4b23af7fdb90721ca3edb7329184d1649"
Jul 28 11:35:30.410393 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:30.410372    2561 scope.go:117] "RemoveContainer" containerID="5d8738ea12755868ece1ec797d12a9caaf1ceb60d19225d1435047b28a2a0cc1"
Jul 28 11:35:30.410572 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:30.410544    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88547" event={"ID":"34712309-48b9-4e19-a7dc-92be37026a55","Type":"ContainerStarted","Data":"1fbcb9656f7705f880d666675418815ce2631faf8fab93233ba81379566ae0cf"}
Jul 28 11:35:30.410668 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:35:30.410627    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx_open-cluster-management-agent-addon(7872f248-f89b-45f1-94fc-70fb87288acf)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx" podUID="7872f248-f89b-45f1-94fc-70fb87288acf"
Jul 28 11:35:32.490406 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:32.490373    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:35:32.492790 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:32.492763    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f978d864-c551-4fcf-8a60-ef424a5556a0-metrics-tls\") pod \"dns-default-f7m7q\" (UID: \"f978d864-c551-4fcf-8a60-ef424a5556a0\") " pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:35:32.708153 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:32.708122    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l5ffw\""
Jul 28 11:35:32.716170 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:32.716146    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:35:35.003251 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:35.003227    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-f7m7q"]
Jul 28 11:35:35.005911 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:35:35.005884    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf978d864_c551_4fcf_8a60_ef424a5556a0.slice/crio-4e0c56b4f40a26642b66fb04fe8975046eb45b743f716cf91da13ec003b75061 WatchSource:0}: Error finding container 4e0c56b4f40a26642b66fb04fe8975046eb45b743f716cf91da13ec003b75061: Status 404 returned error can't find the container with id 4e0c56b4f40a26642b66fb04fe8975046eb45b743f716cf91da13ec003b75061
Jul 28 11:35:35.429758 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:35.429728    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88547" event={"ID":"34712309-48b9-4e19-a7dc-92be37026a55","Type":"ContainerStarted","Data":"437715d545c0bac9bdddd0824874d9fd58431d2408b3e85d82bd6822efe4d3c1"}
Jul 28 11:35:35.430660 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:35.430636    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f7m7q" event={"ID":"f978d864-c551-4fcf-8a60-ef424a5556a0","Type":"ContainerStarted","Data":"4e0c56b4f40a26642b66fb04fe8975046eb45b743f716cf91da13ec003b75061"}
Jul 28 11:35:36.355599 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:36.355550    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx"
Jul 28 11:35:36.356010 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:36.355959    2561 scope.go:117] "RemoveContainer" containerID="5d8738ea12755868ece1ec797d12a9caaf1ceb60d19225d1435047b28a2a0cc1"
Jul 28 11:35:36.356168 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:35:36.356136    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx_open-cluster-management-agent-addon(7872f248-f89b-45f1-94fc-70fb87288acf)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx" podUID="7872f248-f89b-45f1-94fc-70fb87288acf"
Jul 28 11:35:37.863356 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:37.863315    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9" podUID="0b8d7ed2-882c-41b9-ae46-c30e86605e19" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 11:35:38.293558 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.293473    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ssmwx"]
Jul 28 11:35:38.305101 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.305074    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.307459 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.307422    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 11:35:38.308025 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.308000    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gnbvz\""
Jul 28 11:35:38.308153 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.308059    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 11:35:38.308153 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.308072    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 11:35:38.308153 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.308009    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 11:35:38.308313 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.308075    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 11:35:38.308401 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.308382    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 11:35:38.339615 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.339573    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74056816-9582-42e1-a90c-fc0a94603c18-sys\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.339768 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.339655    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74056816-9582-42e1-a90c-fc0a94603c18-metrics-client-ca\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.339768 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.339683    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/74056816-9582-42e1-a90c-fc0a94603c18-node-exporter-textfile\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.339768 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.339706    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/74056816-9582-42e1-a90c-fc0a94603c18-node-exporter-tls\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.339768 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.339734    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/74056816-9582-42e1-a90c-fc0a94603c18-node-exporter-accelerators-collector-config\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.339768 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.339761    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/74056816-9582-42e1-a90c-fc0a94603c18-root\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.339985 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.339789    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74056816-9582-42e1-a90c-fc0a94603c18-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.339985 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.339884    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnd94\" (UniqueName: \"kubernetes.io/projected/74056816-9582-42e1-a90c-fc0a94603c18-kube-api-access-nnd94\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.339985 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.339976    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/74056816-9582-42e1-a90c-fc0a94603c18-node-exporter-wtmp\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.440786 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.440755    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74056816-9582-42e1-a90c-fc0a94603c18-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.440956 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.440794    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnd94\" (UniqueName: \"kubernetes.io/projected/74056816-9582-42e1-a90c-fc0a94603c18-kube-api-access-nnd94\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.440956 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.440842    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/74056816-9582-42e1-a90c-fc0a94603c18-node-exporter-wtmp\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.440956 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.440859    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74056816-9582-42e1-a90c-fc0a94603c18-sys\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.440956 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.440901    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74056816-9582-42e1-a90c-fc0a94603c18-metrics-client-ca\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.440956 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.440926    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/74056816-9582-42e1-a90c-fc0a94603c18-node-exporter-textfile\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.440956 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.440942    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/74056816-9582-42e1-a90c-fc0a94603c18-node-exporter-tls\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.441278 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.440962    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/74056816-9582-42e1-a90c-fc0a94603c18-node-exporter-accelerators-collector-config\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.441278 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.440991    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/74056816-9582-42e1-a90c-fc0a94603c18-root\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.441278 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.441038    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/74056816-9582-42e1-a90c-fc0a94603c18-node-exporter-wtmp\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.441426 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.441306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/74056816-9582-42e1-a90c-fc0a94603c18-node-exporter-textfile\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.441426 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.441372    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74056816-9582-42e1-a90c-fc0a94603c18-sys\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.441426 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.441418    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/74056816-9582-42e1-a90c-fc0a94603c18-root\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.441704 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.441657    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74056816-9582-42e1-a90c-fc0a94603c18-metrics-client-ca\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.441832 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.441732    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/74056816-9582-42e1-a90c-fc0a94603c18-node-exporter-accelerators-collector-config\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.443330 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.443309    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74056816-9582-42e1-a90c-fc0a94603c18-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.443429 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.443409    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/74056816-9582-42e1-a90c-fc0a94603c18-node-exporter-tls\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.451185 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.451158    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnd94\" (UniqueName: \"kubernetes.io/projected/74056816-9582-42e1-a90c-fc0a94603c18-kube-api-access-nnd94\") pod \"node-exporter-ssmwx\" (UID: \"74056816-9582-42e1-a90c-fc0a94603c18\") " pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.616191 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:38.616133    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ssmwx"
Jul 28 11:35:38.682871 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:35:38.682841    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74056816_9582_42e1_a90c_fc0a94603c18.slice/crio-8b24c4a47dbaf785fe7d2a1de61ece4faa8987ed90719c0fc62375b4bc3191b1 WatchSource:0}: Error finding container 8b24c4a47dbaf785fe7d2a1de61ece4faa8987ed90719c0fc62375b4bc3191b1: Status 404 returned error can't find the container with id 8b24c4a47dbaf785fe7d2a1de61ece4faa8987ed90719c0fc62375b4bc3191b1
Jul 28 11:35:39.447470 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:39.446798    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f7m7q" event={"ID":"f978d864-c551-4fcf-8a60-ef424a5556a0","Type":"ContainerStarted","Data":"4518fd070bc4c0dc37277c4cbf602a8f1136d63a271171e10cf7257e1fb229f5"}
Jul 28 11:35:39.447470 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:39.446864    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-f7m7q" event={"ID":"f978d864-c551-4fcf-8a60-ef424a5556a0","Type":"ContainerStarted","Data":"3d9f76e6f1f4561c9962ccbae203a1a39b08656eb246bffafd9b430c87346d41"}
Jul 28 11:35:39.447470 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:39.446894    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:35:39.448483 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:39.448425    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ssmwx" event={"ID":"74056816-9582-42e1-a90c-fc0a94603c18","Type":"ContainerStarted","Data":"8b24c4a47dbaf785fe7d2a1de61ece4faa8987ed90719c0fc62375b4bc3191b1"}
Jul 28 11:35:39.451132 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:39.451107    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88547" event={"ID":"34712309-48b9-4e19-a7dc-92be37026a55","Type":"ContainerStarted","Data":"13c741c9300311a7e1526c9bc2f2cef69099dcd5dc3a4f01b6e1d233fb03cb37"}
Jul 28 11:35:39.463473 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:39.463428    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-f7m7q" podStartSLOduration=253.792304397 podStartE2EDuration="4m17.463410797s" totalImagesPullingTime="3.6711064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:31:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:35:39.46308885 +0000 UTC m=+291.321744845" watchObservedRunningTime="2026-07-28 11:35:39.463410797 +0000 UTC m=+291.322066800"
Jul 28 11:35:40.459132 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:40.459097    2561 generic.go:356] "Generic (PLEG): container finished" podID="74056816-9582-42e1-a90c-fc0a94603c18" containerID="25fa61f432adf8f2b4efda01901e8163371a61fb0b5626b6c493374608eba2ef" exitCode=0
Jul 28 11:35:40.459620 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:40.459190    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ssmwx" event={"ID":"74056816-9582-42e1-a90c-fc0a94603c18","Type":"ContainerDied","Data":"25fa61f432adf8f2b4efda01901e8163371a61fb0b5626b6c493374608eba2ef"}
Jul 28 11:35:41.465851 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:41.465795    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ssmwx" event={"ID":"74056816-9582-42e1-a90c-fc0a94603c18","Type":"ContainerStarted","Data":"eebf39aa0b936215c1824cdfd8427856e05a2dab6e5ff9909e333b4b11646952"}
Jul 28 11:35:42.470857 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:42.470774    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ssmwx" event={"ID":"74056816-9582-42e1-a90c-fc0a94603c18","Type":"ContainerStarted","Data":"623d9d7d52e014cac519c0139fc12bb323bc40f115058ad6b43a6144d6875196"}
Jul 28 11:35:42.472518 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:42.472496    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88547" event={"ID":"34712309-48b9-4e19-a7dc-92be37026a55","Type":"ContainerStarted","Data":"543c71832b376073702c83ac5b412ce80f554749c1a4c87f9d6981e8c8736694"}
Jul 28 11:35:42.489242 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:42.489200    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ssmwx" podStartSLOduration=3.301698655 podStartE2EDuration="4.489188988s" totalImagesPullingTime="1.115608193s" totalInitContainerRuntime="71.88214ms" isStatefulPod=true podCreationTimestamp="2026-07-28 11:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:35:42.487873531 +0000 UTC m=+294.346529532" watchObservedRunningTime="2026-07-28 11:35:42.489188988 +0000 UTC m=+294.347844986"
Jul 28 11:35:42.505439 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:42.505394    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-88547" podStartSLOduration=33.533958704 podStartE2EDuration="45.505381174s" totalImagesPullingTime="11.97142247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 11:34:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:35:42.505313367 +0000 UTC m=+294.363969366" watchObservedRunningTime="2026-07-28 11:35:42.505381174 +0000 UTC m=+294.364037175"
Jul 28 11:35:42.541603 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:35:42.541563    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ss7h8" podUID="85d71f38-a39a-4a1c-90dc-f5b941567052"
Jul 28 11:35:42.921041 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:42.921007    2561 patch_prober.go:28] interesting pod/image-registry-5c46895c78-mnn2v container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 28 11:35:42.921202 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:42.921070    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5c46895c78-mnn2v" podUID="0b4b1d25-a678-4bfa-8320-d0210806a661" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 11:35:44.395104 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.395073    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c46895c78-mnn2v"
Jul 28 11:35:44.532494 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.532453    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 11:35:44.536004 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.535988    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.538484 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.538464    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 11:35:44.538609 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.538535    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 11:35:44.538609 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.538537    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 11:35:44.538861 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.538844    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 11:35:44.538931 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.538919    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 11:35:44.538992 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.538963    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 11:35:44.539045 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.538964    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 11:35:44.539045 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.539030    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 11:35:44.539146 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.538970    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 11:35:44.539146 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.539050    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 11:35:44.539318 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.539304    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 11:35:44.539388 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.539351    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5trqb\""
Jul 28 11:35:44.541042 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.541014    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 11:35:44.541354 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.541336    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5k9ojc4abelm3\""
Jul 28 11:35:44.541546 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.541512    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 11:35:44.541640 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.541513    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 11:35:44.543995 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.543972    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 11:35:44.554109 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.554087    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 11:35:44.589726 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.589699    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.589835 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.589732    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.589835 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.589753    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.589835 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.589768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.589835 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.589783    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-web-config\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.589835 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.589825    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zr52\" (UniqueName: \"kubernetes.io/projected/3286bec2-8f47-4cf0-8894-c6b888cc1be5-kube-api-access-7zr52\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.590004 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.589881    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.590004 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.589903    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.590004 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.589969    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.590004 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.589993    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.590121 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.590036    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.590121 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.590064    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.590121 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.590092    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.590121 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.590117    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.590233 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.590156    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.590233 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.590177    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.590233 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.590194    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-config\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.590233 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.590219    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.590340 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.590261    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3286bec2-8f47-4cf0-8894-c6b888cc1be5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.590340 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.590284    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3286bec2-8f47-4cf0-8894-c6b888cc1be5-config-out\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.691624 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.691559    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.691624 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.691604    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.691624 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.691621    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.691821 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.691642    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.691821 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.691662    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.691821 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.691680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-config\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.691821 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.691701    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.691821 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.691727    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3286bec2-8f47-4cf0-8894-c6b888cc1be5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.691821 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.691748    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3286bec2-8f47-4cf0-8894-c6b888cc1be5-config-out\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.691821 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.691769    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.691821 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.691795    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.691821 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.691820    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.692604 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.692411    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.692604 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.692452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-web-config\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.692604 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.692483    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zr52\" (UniqueName: \"kubernetes.io/projected/3286bec2-8f47-4cf0-8894-c6b888cc1be5-kube-api-access-7zr52\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.692604 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.692499    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.692604 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.692513    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.692604 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.692537    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.692986 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.692631    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.692986 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.692664    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.692986 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.692693    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.693331 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.693304    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.695697 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.695005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.695697 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.695422    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.695697 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.695429    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.695697 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.695430    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-config\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.697141 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.695733    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.697141 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.695928    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.697141 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.696066    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.697141 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.696187    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.697141 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.696356    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.697141 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.696357    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.697141 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.696653    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.697141 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.696705    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3286bec2-8f47-4cf0-8894-c6b888cc1be5-config-out\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.697141 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.696844    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.697141 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.696924    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.697849 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.697824    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.697988 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.697970    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3286bec2-8f47-4cf0-8894-c6b888cc1be5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.698117 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.698098    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-web-config\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.705808 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.705791    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zr52\" (UniqueName: \"kubernetes.io/projected/3286bec2-8f47-4cf0-8894-c6b888cc1be5-kube-api-access-7zr52\") pod \"prometheus-k8s-0\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.847032 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.846999    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:35:44.979306 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:44.979223    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 11:35:44.982436 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:35:44.982413    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3286bec2_8f47_4cf0_8894_c6b888cc1be5.slice/crio-65753ab12ee2dd054ec5444aa979afa8dd288784c2d84cea578ac01e20014c87 WatchSource:0}: Error finding container 65753ab12ee2dd054ec5444aa979afa8dd288784c2d84cea578ac01e20014c87: Status 404 returned error can't find the container with id 65753ab12ee2dd054ec5444aa979afa8dd288784c2d84cea578ac01e20014c87
Jul 28 11:35:45.483565 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:45.483529    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3286bec2-8f47-4cf0-8894-c6b888cc1be5","Type":"ContainerStarted","Data":"65753ab12ee2dd054ec5444aa979afa8dd288784c2d84cea578ac01e20014c87"}
Jul 28 11:35:46.488412 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:46.488375    2561 generic.go:356] "Generic (PLEG): container finished" podID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerID="d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5" exitCode=0
Jul 28 11:35:46.488412 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:46.488413    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3286bec2-8f47-4cf0-8894-c6b888cc1be5","Type":"ContainerDied","Data":"d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5"}
Jul 28 11:35:47.541338 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:47.541302    2561 scope.go:117] "RemoveContainer" containerID="5d8738ea12755868ece1ec797d12a9caaf1ceb60d19225d1435047b28a2a0cc1"
Jul 28 11:35:47.862514 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:47.862479    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9" podUID="0b8d7ed2-882c-41b9-ae46-c30e86605e19" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 11:35:47.862695 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:47.862550    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9"
Jul 28 11:35:47.863039 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:47.863010    2561 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"495201c25a55fe7b74141368e8cb954cfa8f4c28ae07b687d9c79161fdcaac43"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 28 11:35:47.863087 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:47.863057    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9" podUID="0b8d7ed2-882c-41b9-ae46-c30e86605e19" containerName="service-proxy" containerID="cri-o://495201c25a55fe7b74141368e8cb954cfa8f4c28ae07b687d9c79161fdcaac43" gracePeriod=30
Jul 28 11:35:48.457609 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:48.457566    2561 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 11:35:48.497116 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:48.497078    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75bb9c9bb-l9bqx" event={"ID":"7872f248-f89b-45f1-94fc-70fb87288acf","Type":"ContainerStarted","Data":"87537e596d8d18b095c9df6609e7052c13f06436b08ecf583597e70af1335d6b"}
Jul 28 11:35:48.499563 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:48.499533    2561 generic.go:356] "Generic (PLEG): container finished" podID="0b8d7ed2-882c-41b9-ae46-c30e86605e19" containerID="495201c25a55fe7b74141368e8cb954cfa8f4c28ae07b687d9c79161fdcaac43" exitCode=2
Jul 28 11:35:48.499705 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:48.499613    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9" event={"ID":"0b8d7ed2-882c-41b9-ae46-c30e86605e19","Type":"ContainerDied","Data":"495201c25a55fe7b74141368e8cb954cfa8f4c28ae07b687d9c79161fdcaac43"}
Jul 28 11:35:48.499705 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:48.499660    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8697b44576-v5br9" event={"ID":"0b8d7ed2-882c-41b9-ae46-c30e86605e19","Type":"ContainerStarted","Data":"b1f9fd66db14fadf0b759e27e58829bc6246532cfe6f27e6b25cac40b8976dfc"}
Jul 28 11:35:49.462260 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:49.462228    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-f7m7q"
Jul 28 11:35:50.008288 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:50.008263    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 11:35:50.509900 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:50.509866    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3286bec2-8f47-4cf0-8894-c6b888cc1be5","Type":"ContainerStarted","Data":"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922"}
Jul 28 11:35:50.510298 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:50.509950    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3286bec2-8f47-4cf0-8894-c6b888cc1be5","Type":"ContainerStarted","Data":"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff"}
Jul 28 11:35:52.520267 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:52.520236    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3286bec2-8f47-4cf0-8894-c6b888cc1be5","Type":"ContainerStarted","Data":"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf"}
Jul 28 11:35:52.520643 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:52.520301    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3286bec2-8f47-4cf0-8894-c6b888cc1be5","Type":"ContainerStarted","Data":"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b"}
Jul 28 11:35:52.520643 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:52.520332    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3286bec2-8f47-4cf0-8894-c6b888cc1be5","Type":"ContainerStarted","Data":"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597"}
Jul 28 11:35:52.520643 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:52.520356    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3286bec2-8f47-4cf0-8894-c6b888cc1be5","Type":"ContainerStarted","Data":"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288"}
Jul 28 11:35:52.556190 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:52.556138    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.576609327 podStartE2EDuration="8.556122577s" totalImagesPullingTime="5.956747515s" totalInitContainerRuntime="22.765735ms" isStatefulPod=false podCreationTimestamp="2026-07-28 11:35:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:35:52.554642872 +0000 UTC m=+304.413298871" watchObservedRunningTime="2026-07-28 11:35:52.556122577 +0000 UTC m=+304.414778578"
Jul 28 11:35:53.541111 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:53.541082    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:35:54.847959 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:35:54.847916    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:36:44.847997 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:36:44.847920    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:36:44.865996 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:36:44.865971    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:36:45.709894 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:36:45.709869    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:02.386354 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:02.386316    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs\") pod \"network-metrics-daemon-ss7h8\" (UID: \"85d71f38-a39a-4a1c-90dc-f5b941567052\") " pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:37:02.388523 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:02.388502    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85d71f38-a39a-4a1c-90dc-f5b941567052-metrics-certs\") pod \"network-metrics-daemon-ss7h8\" (UID: \"85d71f38-a39a-4a1c-90dc-f5b941567052\") " pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:37:02.543790 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:02.543764    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2mvc2\""
Jul 28 11:37:02.552325 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:02.552304    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ss7h8"
Jul 28 11:37:02.697769 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:02.697734    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ss7h8"]
Jul 28 11:37:02.700672 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:37:02.700648    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d71f38_a39a_4a1c_90dc_f5b941567052.slice/crio-cdeae96c705bf181573eb9e482ab76934d0315cad13ca34796bb368122eba2f6 WatchSource:0}: Error finding container cdeae96c705bf181573eb9e482ab76934d0315cad13ca34796bb368122eba2f6: Status 404 returned error can't find the container with id cdeae96c705bf181573eb9e482ab76934d0315cad13ca34796bb368122eba2f6
Jul 28 11:37:02.749559 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:02.749531    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ss7h8" event={"ID":"85d71f38-a39a-4a1c-90dc-f5b941567052","Type":"ContainerStarted","Data":"cdeae96c705bf181573eb9e482ab76934d0315cad13ca34796bb368122eba2f6"}
Jul 28 11:37:02.967624 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:02.967507    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 11:37:02.967899 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:02.967865    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="kube-rbac-proxy-web" containerID="cri-o://6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597" gracePeriod=600
Jul 28 11:37:02.968515 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:02.967875    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="kube-rbac-proxy" containerID="cri-o://4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b" gracePeriod=600
Jul 28 11:37:02.968515 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:02.967864    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="prometheus" containerID="cri-o://4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff" gracePeriod=600
Jul 28 11:37:02.968515 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:02.968228    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="config-reloader" containerID="cri-o://47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922" gracePeriod=600
Jul 28 11:37:02.968515 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:02.967877    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf" gracePeriod=600
Jul 28 11:37:02.968788 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:02.968739    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="thanos-sidecar" containerID="cri-o://1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288" gracePeriod=600
Jul 28 11:37:03.224400 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.224342    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.293686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.293659    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-metrics-client-certs\") pod \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") "
Jul 28 11:37:03.293686 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.293690    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3286bec2-8f47-4cf0-8894-c6b888cc1be5-tls-assets\") pod \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") "
Jul 28 11:37:03.293903 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.293718    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-prometheus-k8s-tls\") pod \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") "
Jul 28 11:37:03.293903 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.293741    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-configmap-kubelet-serving-ca-bundle\") pod \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") "
Jul 28 11:37:03.293903 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.293882    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zr52\" (UniqueName: \"kubernetes.io/projected/3286bec2-8f47-4cf0-8894-c6b888cc1be5-kube-api-access-7zr52\") pod \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") "
Jul 28 11:37:03.294108 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.293916    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-db\") pod \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") "
Jul 28 11:37:03.294108 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.293949    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") "
Jul 28 11:37:03.294108 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.293974    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-rulefiles-1\") pod \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") "
Jul 28 11:37:03.294108 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.294017    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-rulefiles-2\") pod \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") "
Jul 28 11:37:03.294108 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.294047    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3286bec2-8f47-4cf0-8894-c6b888cc1be5-config-out\") pod \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") "
Jul 28 11:37:03.294108 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.294073    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-kube-rbac-proxy\") pod \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") "
Jul 28 11:37:03.294108 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.294092    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "3286bec2-8f47-4cf0-8894-c6b888cc1be5" (UID: "3286bec2-8f47-4cf0-8894-c6b888cc1be5"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:37:03.294452 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.294100    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-web-config\") pod \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") "
Jul 28 11:37:03.294452 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.294157    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-configmap-serving-certs-ca-bundle\") pod \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") "
Jul 28 11:37:03.294452 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.294188    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-thanos-prometheus-http-client-file\") pod \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") "
Jul 28 11:37:03.294452 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.294346    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-grpc-tls\") pod \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") "
Jul 28 11:37:03.294452 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.294377    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-config\") pod \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") "
Jul 28 11:37:03.294452 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.294408    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-trusted-ca-bundle\") pod \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") "
Jul 28 11:37:03.294452 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.294415    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "3286bec2-8f47-4cf0-8894-c6b888cc1be5" (UID: "3286bec2-8f47-4cf0-8894-c6b888cc1be5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:37:03.294816 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.294461    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") "
Jul 28 11:37:03.294816 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.294501    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-configmap-metrics-client-ca\") pod \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") "
Jul 28 11:37:03.294816 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.294532    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-rulefiles-0\") pod \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\" (UID: \"3286bec2-8f47-4cf0-8894-c6b888cc1be5\") "
Jul 28 11:37:03.294816 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.294767    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:37:03.294816 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.294788    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:37:03.296221 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.295318    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "3286bec2-8f47-4cf0-8894-c6b888cc1be5" (UID: "3286bec2-8f47-4cf0-8894-c6b888cc1be5"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:37:03.296344 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.296119    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "3286bec2-8f47-4cf0-8894-c6b888cc1be5" (UID: "3286bec2-8f47-4cf0-8894-c6b888cc1be5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:37:03.296344 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.296304    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "3286bec2-8f47-4cf0-8894-c6b888cc1be5" (UID: "3286bec2-8f47-4cf0-8894-c6b888cc1be5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:37:03.296556 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.296391    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "3286bec2-8f47-4cf0-8894-c6b888cc1be5" (UID: "3286bec2-8f47-4cf0-8894-c6b888cc1be5"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:37:03.296694 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.296672    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "3286bec2-8f47-4cf0-8894-c6b888cc1be5" (UID: "3286bec2-8f47-4cf0-8894-c6b888cc1be5"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:37:03.296852 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.296831    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "3286bec2-8f47-4cf0-8894-c6b888cc1be5" (UID: "3286bec2-8f47-4cf0-8894-c6b888cc1be5"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:37:03.297251 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.297133    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "3286bec2-8f47-4cf0-8894-c6b888cc1be5" (UID: "3286bec2-8f47-4cf0-8894-c6b888cc1be5"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:37:03.297251 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.297211    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "3286bec2-8f47-4cf0-8894-c6b888cc1be5" (UID: "3286bec2-8f47-4cf0-8894-c6b888cc1be5"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:37:03.299642 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.299605    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "3286bec2-8f47-4cf0-8894-c6b888cc1be5" (UID: "3286bec2-8f47-4cf0-8894-c6b888cc1be5"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:37:03.299918 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.299811    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3286bec2-8f47-4cf0-8894-c6b888cc1be5-kube-api-access-7zr52" (OuterVolumeSpecName: "kube-api-access-7zr52") pod "3286bec2-8f47-4cf0-8894-c6b888cc1be5" (UID: "3286bec2-8f47-4cf0-8894-c6b888cc1be5"). InnerVolumeSpecName "kube-api-access-7zr52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:37:03.299918 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.299878    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-config" (OuterVolumeSpecName: "config") pod "3286bec2-8f47-4cf0-8894-c6b888cc1be5" (UID: "3286bec2-8f47-4cf0-8894-c6b888cc1be5"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:37:03.299918 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.299884    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3286bec2-8f47-4cf0-8894-c6b888cc1be5-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3286bec2-8f47-4cf0-8894-c6b888cc1be5" (UID: "3286bec2-8f47-4cf0-8894-c6b888cc1be5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:37:03.300108 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.299934    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "3286bec2-8f47-4cf0-8894-c6b888cc1be5" (UID: "3286bec2-8f47-4cf0-8894-c6b888cc1be5"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:37:03.300162 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.300124    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3286bec2-8f47-4cf0-8894-c6b888cc1be5-config-out" (OuterVolumeSpecName: "config-out") pod "3286bec2-8f47-4cf0-8894-c6b888cc1be5" (UID: "3286bec2-8f47-4cf0-8894-c6b888cc1be5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:37:03.300406 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.300312    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "3286bec2-8f47-4cf0-8894-c6b888cc1be5" (UID: "3286bec2-8f47-4cf0-8894-c6b888cc1be5"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:37:03.300406 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.300377    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "3286bec2-8f47-4cf0-8894-c6b888cc1be5" (UID: "3286bec2-8f47-4cf0-8894-c6b888cc1be5"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:37:03.301412 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.301379    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "3286bec2-8f47-4cf0-8894-c6b888cc1be5" (UID: "3286bec2-8f47-4cf0-8894-c6b888cc1be5"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:37:03.309067 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.309041    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-web-config" (OuterVolumeSpecName: "web-config") pod "3286bec2-8f47-4cf0-8894-c6b888cc1be5" (UID: "3286bec2-8f47-4cf0-8894-c6b888cc1be5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:37:03.396095 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.396066    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:37:03.396462 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.396100    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-configmap-metrics-client-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:37:03.396462 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.396115    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:37:03.396462 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.396128    2561 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-metrics-client-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:37:03.396462 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.396142    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3286bec2-8f47-4cf0-8894-c6b888cc1be5-tls-assets\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:37:03.396462 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.396155    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:37:03.396462 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.396169    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zr52\" (UniqueName: \"kubernetes.io/projected/3286bec2-8f47-4cf0-8894-c6b888cc1be5-kube-api-access-7zr52\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:37:03.396462 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.396193    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-db\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:37:03.396462 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.396204    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:37:03.396462 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.396217    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:37:03.396462 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.396229    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3286bec2-8f47-4cf0-8894-c6b888cc1be5-config-out\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:37:03.396462 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.396241    2561 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-kube-rbac-proxy\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:37:03.396462 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.396253    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-web-config\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:37:03.396462 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.396262    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:37:03.396462 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.396273    2561 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:37:03.396462 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.396284    2561 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-secret-grpc-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:37:03.396462 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.396293    2561 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/3286bec2-8f47-4cf0-8894-c6b888cc1be5-config\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:37:03.396462 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.396306    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3286bec2-8f47-4cf0-8894-c6b888cc1be5-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:37:03.755519 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.755484    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ss7h8" event={"ID":"85d71f38-a39a-4a1c-90dc-f5b941567052","Type":"ContainerStarted","Data":"7c252b4137d0e8a4eaf5f3923fc97d379707ad642217aa4f6b051eb35f704972"}
Jul 28 11:37:03.755667 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.755540    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ss7h8" event={"ID":"85d71f38-a39a-4a1c-90dc-f5b941567052","Type":"ContainerStarted","Data":"e3bd0ff3063e49683cb36b11017d78667d5f50803c8fe23ab173f745e7d6153a"}
Jul 28 11:37:03.758492 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.758467    2561 generic.go:356] "Generic (PLEG): container finished" podID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerID="a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf" exitCode=0
Jul 28 11:37:03.758492 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.758491    2561 generic.go:356] "Generic (PLEG): container finished" podID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerID="4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b" exitCode=0
Jul 28 11:37:03.758669 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.758498    2561 generic.go:356] "Generic (PLEG): container finished" podID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerID="6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597" exitCode=0
Jul 28 11:37:03.758669 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.758506    2561 generic.go:356] "Generic (PLEG): container finished" podID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerID="1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288" exitCode=0
Jul 28 11:37:03.758669 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.758511    2561 generic.go:356] "Generic (PLEG): container finished" podID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerID="47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922" exitCode=0
Jul 28 11:37:03.758669 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.758516    2561 generic.go:356] "Generic (PLEG): container finished" podID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerID="4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff" exitCode=0
Jul 28 11:37:03.758669 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.758544    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3286bec2-8f47-4cf0-8894-c6b888cc1be5","Type":"ContainerDied","Data":"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf"}
Jul 28 11:37:03.758669 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.758554    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.758669 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.758653    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3286bec2-8f47-4cf0-8894-c6b888cc1be5","Type":"ContainerDied","Data":"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b"}
Jul 28 11:37:03.758994 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.758703    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3286bec2-8f47-4cf0-8894-c6b888cc1be5","Type":"ContainerDied","Data":"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597"}
Jul 28 11:37:03.758994 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.758715    2561 scope.go:117] "RemoveContainer" containerID="a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf"
Jul 28 11:37:03.758994 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.758749    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3286bec2-8f47-4cf0-8894-c6b888cc1be5","Type":"ContainerDied","Data":"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288"}
Jul 28 11:37:03.758994 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.758791    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3286bec2-8f47-4cf0-8894-c6b888cc1be5","Type":"ContainerDied","Data":"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922"}
Jul 28 11:37:03.758994 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.758833    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3286bec2-8f47-4cf0-8894-c6b888cc1be5","Type":"ContainerDied","Data":"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff"}
Jul 28 11:37:03.758994 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.758876    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3286bec2-8f47-4cf0-8894-c6b888cc1be5","Type":"ContainerDied","Data":"65753ab12ee2dd054ec5444aa979afa8dd288784c2d84cea578ac01e20014c87"}
Jul 28 11:37:03.765354 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.765339    2561 scope.go:117] "RemoveContainer" containerID="4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b"
Jul 28 11:37:03.770878 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.770832    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ss7h8" podStartSLOduration=374.941459079 podStartE2EDuration="6m15.770815666s" totalImagesPullingTime="829.356587ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:37:03.770430141 +0000 UTC m=+375.629086141" watchObservedRunningTime="2026-07-28 11:37:03.770815666 +0000 UTC m=+375.629471667"
Jul 28 11:37:03.776704 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.776689    2561 scope.go:117] "RemoveContainer" containerID="6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597"
Jul 28 11:37:03.782623 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.782598    2561 scope.go:117] "RemoveContainer" containerID="1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288"
Jul 28 11:37:03.786332 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.786311    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 11:37:03.789488 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.789471    2561 scope.go:117] "RemoveContainer" containerID="47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922"
Jul 28 11:37:03.790089 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.790068    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 11:37:03.795479 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.795464    2561 scope.go:117] "RemoveContainer" containerID="4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff"
Jul 28 11:37:03.801951 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.801935    2561 scope.go:117] "RemoveContainer" containerID="d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5"
Jul 28 11:37:03.807883 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.807867    2561 scope.go:117] "RemoveContainer" containerID="a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf"
Jul 28 11:37:03.808116 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:37:03.808099    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf\": container with ID starting with a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf not found: ID does not exist" containerID="a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf"
Jul 28 11:37:03.808159 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.808122    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf"} err="failed to get container status \"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf\": rpc error: code = NotFound desc = could not find container \"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf\": container with ID starting with a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf not found: ID does not exist"
Jul 28 11:37:03.808159 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.808139    2561 scope.go:117] "RemoveContainer" containerID="4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b"
Jul 28 11:37:03.808332 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:37:03.808317    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b\": container with ID starting with 4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b not found: ID does not exist" containerID="4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b"
Jul 28 11:37:03.808373 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.808335    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b"} err="failed to get container status \"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b\": rpc error: code = NotFound desc = could not find container \"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b\": container with ID starting with 4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b not found: ID does not exist"
Jul 28 11:37:03.808373 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.808347    2561 scope.go:117] "RemoveContainer" containerID="6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597"
Jul 28 11:37:03.808545 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:37:03.808529    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597\": container with ID starting with 6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597 not found: ID does not exist" containerID="6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597"
Jul 28 11:37:03.808602 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.808551    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597"} err="failed to get container status \"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597\": rpc error: code = NotFound desc = could not find container \"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597\": container with ID starting with 6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597 not found: ID does not exist"
Jul 28 11:37:03.808602 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.808566    2561 scope.go:117] "RemoveContainer" containerID="1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288"
Jul 28 11:37:03.808812 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:37:03.808796    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288\": container with ID starting with 1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288 not found: ID does not exist" containerID="1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288"
Jul 28 11:37:03.808859 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.808815    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288"} err="failed to get container status \"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288\": rpc error: code = NotFound desc = could not find container \"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288\": container with ID starting with 1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288 not found: ID does not exist"
Jul 28 11:37:03.808859 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.808830    2561 scope.go:117] "RemoveContainer" containerID="47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922"
Jul 28 11:37:03.809093 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:37:03.809069    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922\": container with ID starting with 47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922 not found: ID does not exist" containerID="47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922"
Jul 28 11:37:03.809136 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.809098    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922"} err="failed to get container status \"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922\": rpc error: code = NotFound desc = could not find container \"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922\": container with ID starting with 47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922 not found: ID does not exist"
Jul 28 11:37:03.809136 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.809119    2561 scope.go:117] "RemoveContainer" containerID="4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff"
Jul 28 11:37:03.809332 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:37:03.809314    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff\": container with ID starting with 4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff not found: ID does not exist" containerID="4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff"
Jul 28 11:37:03.809366 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.809337    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff"} err="failed to get container status \"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff\": rpc error: code = NotFound desc = could not find container \"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff\": container with ID starting with 4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff not found: ID does not exist"
Jul 28 11:37:03.809366 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.809349    2561 scope.go:117] "RemoveContainer" containerID="d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5"
Jul 28 11:37:03.809553 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:37:03.809538    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5\": container with ID starting with d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5 not found: ID does not exist" containerID="d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5"
Jul 28 11:37:03.809624 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.809555    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5"} err="failed to get container status \"d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5\": rpc error: code = NotFound desc = could not find container \"d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5\": container with ID starting with d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5 not found: ID does not exist"
Jul 28 11:37:03.809624 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.809568    2561 scope.go:117] "RemoveContainer" containerID="a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf"
Jul 28 11:37:03.809811 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.809793    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf"} err="failed to get container status \"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf\": rpc error: code = NotFound desc = could not find container \"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf\": container with ID starting with a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf not found: ID does not exist"
Jul 28 11:37:03.809848 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.809813    2561 scope.go:117] "RemoveContainer" containerID="4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b"
Jul 28 11:37:03.810001 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.809986    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b"} err="failed to get container status \"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b\": rpc error: code = NotFound desc = could not find container \"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b\": container with ID starting with 4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b not found: ID does not exist"
Jul 28 11:37:03.810001 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.810001    2561 scope.go:117] "RemoveContainer" containerID="6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597"
Jul 28 11:37:03.810192 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.810175    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597"} err="failed to get container status \"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597\": rpc error: code = NotFound desc = could not find container \"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597\": container with ID starting with 6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597 not found: ID does not exist"
Jul 28 11:37:03.810229 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.810193    2561 scope.go:117] "RemoveContainer" containerID="1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288"
Jul 28 11:37:03.810360 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.810345    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288"} err="failed to get container status \"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288\": rpc error: code = NotFound desc = could not find container \"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288\": container with ID starting with 1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288 not found: ID does not exist"
Jul 28 11:37:03.810399 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.810361    2561 scope.go:117] "RemoveContainer" containerID="47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922"
Jul 28 11:37:03.810596 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.810562    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922"} err="failed to get container status \"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922\": rpc error: code = NotFound desc = could not find container \"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922\": container with ID starting with 47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922 not found: ID does not exist"
Jul 28 11:37:03.810657 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.810600    2561 scope.go:117] "RemoveContainer" containerID="4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff"
Jul 28 11:37:03.810832 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.810816    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff"} err="failed to get container status \"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff\": rpc error: code = NotFound desc = could not find container \"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff\": container with ID starting with 4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff not found: ID does not exist"
Jul 28 11:37:03.810874 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.810833    2561 scope.go:117] "RemoveContainer" containerID="d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5"
Jul 28 11:37:03.811088 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.811065    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5"} err="failed to get container status \"d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5\": rpc error: code = NotFound desc = could not find container \"d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5\": container with ID starting with d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5 not found: ID does not exist"
Jul 28 11:37:03.811162 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.811091    2561 scope.go:117] "RemoveContainer" containerID="a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf"
Jul 28 11:37:03.811331 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.811314    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf"} err="failed to get container status \"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf\": rpc error: code = NotFound desc = could not find container \"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf\": container with ID starting with a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf not found: ID does not exist"
Jul 28 11:37:03.811407 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.811344    2561 scope.go:117] "RemoveContainer" containerID="4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b"
Jul 28 11:37:03.811538 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.811514    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b"} err="failed to get container status \"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b\": rpc error: code = NotFound desc = could not find container \"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b\": container with ID starting with 4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b not found: ID does not exist"
Jul 28 11:37:03.811538 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.811539    2561 scope.go:117] "RemoveContainer" containerID="6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597"
Jul 28 11:37:03.811823 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.811807    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597"} err="failed to get container status \"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597\": rpc error: code = NotFound desc = could not find container \"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597\": container with ID starting with 6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597 not found: ID does not exist"
Jul 28 11:37:03.811895 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.811825    2561 scope.go:117] "RemoveContainer" containerID="1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288"
Jul 28 11:37:03.812061 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.812042    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288"} err="failed to get container status \"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288\": rpc error: code = NotFound desc = could not find container \"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288\": container with ID starting with 1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288 not found: ID does not exist"
Jul 28 11:37:03.812112 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.812063    2561 scope.go:117] "RemoveContainer" containerID="47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922"
Jul 28 11:37:03.812262 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.812246    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922"} err="failed to get container status \"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922\": rpc error: code = NotFound desc = could not find container \"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922\": container with ID starting with 47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922 not found: ID does not exist"
Jul 28 11:37:03.812306 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.812262    2561 scope.go:117] "RemoveContainer" containerID="4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff"
Jul 28 11:37:03.812438 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.812421    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff"} err="failed to get container status \"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff\": rpc error: code = NotFound desc = could not find container \"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff\": container with ID starting with 4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff not found: ID does not exist"
Jul 28 11:37:03.812497 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.812439    2561 scope.go:117] "RemoveContainer" containerID="d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5"
Jul 28 11:37:03.812652 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.812635    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5"} err="failed to get container status \"d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5\": rpc error: code = NotFound desc = could not find container \"d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5\": container with ID starting with d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5 not found: ID does not exist"
Jul 28 11:37:03.812755 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.812653    2561 scope.go:117] "RemoveContainer" containerID="a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf"
Jul 28 11:37:03.812835 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.812821    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf"} err="failed to get container status \"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf\": rpc error: code = NotFound desc = could not find container \"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf\": container with ID starting with a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf not found: ID does not exist"
Jul 28 11:37:03.812890 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.812835    2561 scope.go:117] "RemoveContainer" containerID="4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b"
Jul 28 11:37:03.813056 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.813038    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b"} err="failed to get container status \"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b\": rpc error: code = NotFound desc = could not find container \"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b\": container with ID starting with 4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b not found: ID does not exist"
Jul 28 11:37:03.813099 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.813057    2561 scope.go:117] "RemoveContainer" containerID="6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597"
Jul 28 11:37:03.813255 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.813241    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597"} err="failed to get container status \"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597\": rpc error: code = NotFound desc = could not find container \"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597\": container with ID starting with 6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597 not found: ID does not exist"
Jul 28 11:37:03.813300 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.813256    2561 scope.go:117] "RemoveContainer" containerID="1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288"
Jul 28 11:37:03.813463 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.813445    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288"} err="failed to get container status \"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288\": rpc error: code = NotFound desc = could not find container \"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288\": container with ID starting with 1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288 not found: ID does not exist"
Jul 28 11:37:03.813505 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.813464    2561 scope.go:117] "RemoveContainer" containerID="47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922"
Jul 28 11:37:03.813668 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.813653    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922"} err="failed to get container status \"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922\": rpc error: code = NotFound desc = could not find container \"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922\": container with ID starting with 47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922 not found: ID does not exist"
Jul 28 11:37:03.813717 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.813668    2561 scope.go:117] "RemoveContainer" containerID="4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff"
Jul 28 11:37:03.813850 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.813834    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff"} err="failed to get container status \"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff\": rpc error: code = NotFound desc = could not find container \"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff\": container with ID starting with 4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff not found: ID does not exist"
Jul 28 11:37:03.813891 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.813850    2561 scope.go:117] "RemoveContainer" containerID="d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5"
Jul 28 11:37:03.814032 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.814013    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5"} err="failed to get container status \"d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5\": rpc error: code = NotFound desc = could not find container \"d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5\": container with ID starting with d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5 not found: ID does not exist"
Jul 28 11:37:03.814032 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.814031    2561 scope.go:117] "RemoveContainer" containerID="a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf"
Jul 28 11:37:03.814196 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.814177    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf"} err="failed to get container status \"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf\": rpc error: code = NotFound desc = could not find container \"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf\": container with ID starting with a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf not found: ID does not exist"
Jul 28 11:37:03.814262 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.814198    2561 scope.go:117] "RemoveContainer" containerID="4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b"
Jul 28 11:37:03.814402 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.814387    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b"} err="failed to get container status \"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b\": rpc error: code = NotFound desc = could not find container \"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b\": container with ID starting with 4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b not found: ID does not exist"
Jul 28 11:37:03.814402 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.814401    2561 scope.go:117] "RemoveContainer" containerID="6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597"
Jul 28 11:37:03.814628 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.814611    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597"} err="failed to get container status \"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597\": rpc error: code = NotFound desc = could not find container \"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597\": container with ID starting with 6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597 not found: ID does not exist"
Jul 28 11:37:03.814693 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.814629    2561 scope.go:117] "RemoveContainer" containerID="1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288"
Jul 28 11:37:03.814819 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.814803    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288"} err="failed to get container status \"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288\": rpc error: code = NotFound desc = could not find container \"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288\": container with ID starting with 1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288 not found: ID does not exist"
Jul 28 11:37:03.814873 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.814820    2561 scope.go:117] "RemoveContainer" containerID="47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922"
Jul 28 11:37:03.815038 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.815021    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922"} err="failed to get container status \"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922\": rpc error: code = NotFound desc = could not find container \"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922\": container with ID starting with 47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922 not found: ID does not exist"
Jul 28 11:37:03.815109 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.815040    2561 scope.go:117] "RemoveContainer" containerID="4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff"
Jul 28 11:37:03.815270 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.815242    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff"} err="failed to get container status \"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff\": rpc error: code = NotFound desc = could not find container \"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff\": container with ID starting with 4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff not found: ID does not exist"
Jul 28 11:37:03.815270 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.815265    2561 scope.go:117] "RemoveContainer" containerID="d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5"
Jul 28 11:37:03.815458 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.815439    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5"} err="failed to get container status \"d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5\": rpc error: code = NotFound desc = could not find container \"d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5\": container with ID starting with d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5 not found: ID does not exist"
Jul 28 11:37:03.815538 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.815459    2561 scope.go:117] "RemoveContainer" containerID="a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf"
Jul 28 11:37:03.815674 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.815657    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf"} err="failed to get container status \"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf\": rpc error: code = NotFound desc = could not find container \"a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf\": container with ID starting with a3b1a305ba0fb242fc364ff0a7acfd139f0c1f480ac1edcd4174a6267297afbf not found: ID does not exist"
Jul 28 11:37:03.815715 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.815675    2561 scope.go:117] "RemoveContainer" containerID="4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b"
Jul 28 11:37:03.815901 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.815882    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b"} err="failed to get container status \"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b\": rpc error: code = NotFound desc = could not find container \"4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b\": container with ID starting with 4588e2dbef26386227ca66ba7eeae0bb472805a3dbd08dc81c81b9ab8409722b not found: ID does not exist"
Jul 28 11:37:03.815973 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.815902    2561 scope.go:117] "RemoveContainer" containerID="6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597"
Jul 28 11:37:03.816130 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.816115    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597"} err="failed to get container status \"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597\": rpc error: code = NotFound desc = could not find container \"6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597\": container with ID starting with 6bf971d52112e8facfc2c623cb07dfbaf7622dfbc7cf10ddf014b80f018e9597 not found: ID does not exist"
Jul 28 11:37:03.816188 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.816131    2561 scope.go:117] "RemoveContainer" containerID="1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288"
Jul 28 11:37:03.816324 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.816308    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288"} err="failed to get container status \"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288\": rpc error: code = NotFound desc = could not find container \"1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288\": container with ID starting with 1b62ff2466f2be7bf8cacf6b5b18225021ab507632ade054c6b7034c72049288 not found: ID does not exist"
Jul 28 11:37:03.816368 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.816325    2561 scope.go:117] "RemoveContainer" containerID="47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922"
Jul 28 11:37:03.816547 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.816530    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922"} err="failed to get container status \"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922\": rpc error: code = NotFound desc = could not find container \"47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922\": container with ID starting with 47af8ff67805e6a13bb003a941a0f7b18000b1e078586cb37903a4daadd55922 not found: ID does not exist"
Jul 28 11:37:03.816637 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.816548    2561 scope.go:117] "RemoveContainer" containerID="4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff"
Jul 28 11:37:03.816751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.816736    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff"} err="failed to get container status \"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff\": rpc error: code = NotFound desc = could not find container \"4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff\": container with ID starting with 4381d6700cb00a748fa721874a5e645e485202ebd82d916f6ccf1d4ee21434ff not found: ID does not exist"
Jul 28 11:37:03.816814 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.816752    2561 scope.go:117] "RemoveContainer" containerID="d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5"
Jul 28 11:37:03.816945 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.816930    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5"} err="failed to get container status \"d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5\": rpc error: code = NotFound desc = could not find container \"d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5\": container with ID starting with d94fb11ca6d006e7cb1f9eb8121602b1134f470144a7c62c54005b19d4af38e5 not found: ID does not exist"
Jul 28 11:37:03.820201 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820181    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 11:37:03.820381 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820370    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="config-reloader"
Jul 28 11:37:03.820426 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820383    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="config-reloader"
Jul 28 11:37:03.820426 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820399    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="prometheus"
Jul 28 11:37:03.820426 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820407    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="prometheus"
Jul 28 11:37:03.820426 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820417    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="kube-rbac-proxy"
Jul 28 11:37:03.820426 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820425    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="kube-rbac-proxy"
Jul 28 11:37:03.820568 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820437    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="kube-rbac-proxy-web"
Jul 28 11:37:03.820568 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820442    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="kube-rbac-proxy-web"
Jul 28 11:37:03.820568 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820455    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="thanos-sidecar"
Jul 28 11:37:03.820568 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820461    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="thanos-sidecar"
Jul 28 11:37:03.820568 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820466    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="kube-rbac-proxy-thanos"
Jul 28 11:37:03.820568 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820471    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="kube-rbac-proxy-thanos"
Jul 28 11:37:03.820568 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820478    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="init-config-reloader"
Jul 28 11:37:03.820568 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820483    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="init-config-reloader"
Jul 28 11:37:03.820568 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820515    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="thanos-sidecar"
Jul 28 11:37:03.820568 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820521    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="kube-rbac-proxy"
Jul 28 11:37:03.820568 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820528    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="kube-rbac-proxy-thanos"
Jul 28 11:37:03.820568 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820533    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="prometheus"
Jul 28 11:37:03.820568 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820540    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="config-reloader"
Jul 28 11:37:03.820568 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.820544    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" containerName="kube-rbac-proxy-web"
Jul 28 11:37:03.823469 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.823455    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.825411 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.825394    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 11:37:03.825522 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.825504    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 11:37:03.825693 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.825671    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 11:37:03.825916 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.825891    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 11:37:03.826056 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.826042    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 11:37:03.826131 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.826107    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 11:37:03.826186 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.826132    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 11:37:03.826241 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.826207    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 11:37:03.826364 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.826351    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 11:37:03.826522 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.826508    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5k9ojc4abelm3\""
Jul 28 11:37:03.826616 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.826536    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 11:37:03.826616 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.826553    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 11:37:03.826729 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.826629    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5trqb\""
Jul 28 11:37:03.826729 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.826667    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 11:37:03.826830 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.826814    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 11:37:03.829055 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.829036    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 11:37:03.832927 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.832909    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 11:37:03.841423 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.841405    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 11:37:03.900683 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.900663    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.900766 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.900691    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-web-config\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.900766 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.900712    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-config\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.900766 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.900736    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.900875 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.900812    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.900875 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.900845    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.900875 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.900864    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f7xq\" (UniqueName: \"kubernetes.io/projected/9eb1e89f-7334-4d90-b463-6a99faf7230e-kube-api-access-9f7xq\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.900985 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.900888    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.900985 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.900922    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9eb1e89f-7334-4d90-b463-6a99faf7230e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.900985 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.900944    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.900985 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.900963    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.901145 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.901017    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9eb1e89f-7334-4d90-b463-6a99faf7230e-config-out\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.901145 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.901034    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.901145 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.901055    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.901145 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.901079    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9eb1e89f-7334-4d90-b463-6a99faf7230e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.901145 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.901109    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.901312 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.901149    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.901312 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.901185    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.901312 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.901215    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:03.901312 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:03.901235    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002063 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002029    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9eb1e89f-7334-4d90-b463-6a99faf7230e-config-out\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002063 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002066    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002203 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002090    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002203 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002107    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9eb1e89f-7334-4d90-b463-6a99faf7230e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002203 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002122    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002203 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002144    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002203 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002166    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002203 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002186    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002481 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002204    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002481 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002230    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002481 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002352    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-web-config\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002481 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002381    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-config\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002481 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002399    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002481 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002416    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002481 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002433    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002481 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f7xq\" (UniqueName: \"kubernetes.io/projected/9eb1e89f-7334-4d90-b463-6a99faf7230e-kube-api-access-9f7xq\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002481 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002478    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002943 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002502    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9eb1e89f-7334-4d90-b463-6a99faf7230e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002943 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002538    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002943 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002563    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.002943 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.002764    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.003546 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.003082    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.003546 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.003320    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.003546 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.003362    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.003799 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.003775    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.004273 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.004063    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9eb1e89f-7334-4d90-b463-6a99faf7230e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.005784 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.005145    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.005784 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.005745    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.006047 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.006004    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9eb1e89f-7334-4d90-b463-6a99faf7230e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.007188 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.007162    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.007351 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.007171    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.007351 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.007328    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-web-config\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.007826 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.007560    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.007826 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.007717    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.007826 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.007779    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9eb1e89f-7334-4d90-b463-6a99faf7230e-config-out\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.008021 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.007931    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.008145 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.008122    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.008306 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.008290    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9eb1e89f-7334-4d90-b463-6a99faf7230e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.008932 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.008915    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9eb1e89f-7334-4d90-b463-6a99faf7230e-config\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.011823 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.011804    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f7xq\" (UniqueName: \"kubernetes.io/projected/9eb1e89f-7334-4d90-b463-6a99faf7230e-kube-api-access-9f7xq\") pod \"prometheus-k8s-0\" (UID: \"9eb1e89f-7334-4d90-b463-6a99faf7230e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.133515 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.133493    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:37:04.264567 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.264490    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 11:37:04.266871 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:37:04.266844    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb1e89f_7334_4d90_b463_6a99faf7230e.slice/crio-e6173ed329e094eea6cc5698125995d987979f81de857cd4d098c69d52cce75b WatchSource:0}: Error finding container e6173ed329e094eea6cc5698125995d987979f81de857cd4d098c69d52cce75b: Status 404 returned error can't find the container with id e6173ed329e094eea6cc5698125995d987979f81de857cd4d098c69d52cce75b
Jul 28 11:37:04.549541 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.549463    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3286bec2-8f47-4cf0-8894-c6b888cc1be5" path="/var/lib/kubelet/pods/3286bec2-8f47-4cf0-8894-c6b888cc1be5/volumes"
Jul 28 11:37:04.764212 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.764178    2561 generic.go:356] "Generic (PLEG): container finished" podID="9eb1e89f-7334-4d90-b463-6a99faf7230e" containerID="be1fa4767386d56d02fb48dca2b190337f814e28f82b326e4ea39b4324fd3642" exitCode=0
Jul 28 11:37:04.764426 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.764263    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9eb1e89f-7334-4d90-b463-6a99faf7230e","Type":"ContainerDied","Data":"be1fa4767386d56d02fb48dca2b190337f814e28f82b326e4ea39b4324fd3642"}
Jul 28 11:37:04.764426 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:04.764355    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9eb1e89f-7334-4d90-b463-6a99faf7230e","Type":"ContainerStarted","Data":"e6173ed329e094eea6cc5698125995d987979f81de857cd4d098c69d52cce75b"}
Jul 28 11:37:05.771783 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:05.771748    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9eb1e89f-7334-4d90-b463-6a99faf7230e","Type":"ContainerStarted","Data":"9a6381568aa081777f15e018bc205cb359b684f7b2b3a19794b7cbee7f6f3c8c"}
Jul 28 11:37:05.772249 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:05.771825    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9eb1e89f-7334-4d90-b463-6a99faf7230e","Type":"ContainerStarted","Data":"9de0c7d484b7cf01682ffa4d82a05c9b6564823e319e582e6ff1314202739aa6"}
Jul 28 11:37:05.772249 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:05.771875    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9eb1e89f-7334-4d90-b463-6a99faf7230e","Type":"ContainerStarted","Data":"39141c7191e192dfec09c471ee6ccd800f20b6e2dbbb64a01e3e7c4435c41314"}
Jul 28 11:37:05.772249 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:05.771918    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9eb1e89f-7334-4d90-b463-6a99faf7230e","Type":"ContainerStarted","Data":"f5bb287c77de7bd85cdbd9b684300d9e8869fbc01e5f80c816e4f2e06f81737c"}
Jul 28 11:37:05.772249 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:05.771960    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9eb1e89f-7334-4d90-b463-6a99faf7230e","Type":"ContainerStarted","Data":"2c00b58d5ca876c91f324e50e2eb6955a611b2ed9add06da4250ab621c143efe"}
Jul 28 11:37:05.772249 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:05.772000    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9eb1e89f-7334-4d90-b463-6a99faf7230e","Type":"ContainerStarted","Data":"e24282b8ab7597df018d01b03d3768ae5c51b1963c21793b76c4a18e48647197"}
Jul 28 11:37:05.824297 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:05.824239    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.805670765 podStartE2EDuration="2.824220196s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.549431ms" isStatefulPod=false podCreationTimestamp="2026-07-28 11:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:37:05.8233437 +0000 UTC m=+377.681999711" watchObservedRunningTime="2026-07-28 11:37:05.824220196 +0000 UTC m=+377.682876197"
Jul 28 11:37:09.133660 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:37:09.133623    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:38:04.133939 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:38:04.133895    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:38:04.148812 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:38:04.148785    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:38:04.984989 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:38:04.984961    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 11:40:48.670747 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.670719    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"]
Jul 28 11:40:48.673771 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.673756    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"
Jul 28 11:40:48.675885 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.675861    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 28 11:40:48.675885 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.675887    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 28 11:40:48.676070 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.675951    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 28 11:40:48.676209 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.676196    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sw542\""
Jul 28 11:40:48.676209 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.676201    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 28 11:40:48.695407 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.695384    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"]
Jul 28 11:40:48.806220 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.806195    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/463fa1da-5fff-4de1-815b-2f76070d5165-manifests\") pod \"opendatahub-operator-controller-manager-6755d7577-8fmcq\" (UID: \"463fa1da-5fff-4de1-815b-2f76070d5165\") " pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"
Jul 28 11:40:48.806338 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.806235    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6wwj\" (UniqueName: \"kubernetes.io/projected/463fa1da-5fff-4de1-815b-2f76070d5165-kube-api-access-f6wwj\") pod \"opendatahub-operator-controller-manager-6755d7577-8fmcq\" (UID: \"463fa1da-5fff-4de1-815b-2f76070d5165\") " pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"
Jul 28 11:40:48.806338 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.806264    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/463fa1da-5fff-4de1-815b-2f76070d5165-tmp\") pod \"opendatahub-operator-controller-manager-6755d7577-8fmcq\" (UID: \"463fa1da-5fff-4de1-815b-2f76070d5165\") " pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"
Jul 28 11:40:48.806338 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.806309    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/463fa1da-5fff-4de1-815b-2f76070d5165-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6755d7577-8fmcq\" (UID: \"463fa1da-5fff-4de1-815b-2f76070d5165\") " pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"
Jul 28 11:40:48.806338 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.806328    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/463fa1da-5fff-4de1-815b-2f76070d5165-webhook-cert\") pod \"opendatahub-operator-controller-manager-6755d7577-8fmcq\" (UID: \"463fa1da-5fff-4de1-815b-2f76070d5165\") " pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"
Jul 28 11:40:48.907290 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.907260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/463fa1da-5fff-4de1-815b-2f76070d5165-manifests\") pod \"opendatahub-operator-controller-manager-6755d7577-8fmcq\" (UID: \"463fa1da-5fff-4de1-815b-2f76070d5165\") " pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"
Jul 28 11:40:48.907443 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.907295    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6wwj\" (UniqueName: \"kubernetes.io/projected/463fa1da-5fff-4de1-815b-2f76070d5165-kube-api-access-f6wwj\") pod \"opendatahub-operator-controller-manager-6755d7577-8fmcq\" (UID: \"463fa1da-5fff-4de1-815b-2f76070d5165\") " pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"
Jul 28 11:40:48.907443 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.907322    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/463fa1da-5fff-4de1-815b-2f76070d5165-tmp\") pod \"opendatahub-operator-controller-manager-6755d7577-8fmcq\" (UID: \"463fa1da-5fff-4de1-815b-2f76070d5165\") " pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"
Jul 28 11:40:48.907443 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.907344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/463fa1da-5fff-4de1-815b-2f76070d5165-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6755d7577-8fmcq\" (UID: \"463fa1da-5fff-4de1-815b-2f76070d5165\") " pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"
Jul 28 11:40:48.907443 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.907363    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/463fa1da-5fff-4de1-815b-2f76070d5165-webhook-cert\") pod \"opendatahub-operator-controller-manager-6755d7577-8fmcq\" (UID: \"463fa1da-5fff-4de1-815b-2f76070d5165\") " pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"
Jul 28 11:40:48.907680 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.907661    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/463fa1da-5fff-4de1-815b-2f76070d5165-manifests\") pod \"opendatahub-operator-controller-manager-6755d7577-8fmcq\" (UID: \"463fa1da-5fff-4de1-815b-2f76070d5165\") " pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"
Jul 28 11:40:48.909717 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.909686    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/463fa1da-5fff-4de1-815b-2f76070d5165-tmp\") pod \"opendatahub-operator-controller-manager-6755d7577-8fmcq\" (UID: \"463fa1da-5fff-4de1-815b-2f76070d5165\") " pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"
Jul 28 11:40:48.909830 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.909787    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/463fa1da-5fff-4de1-815b-2f76070d5165-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6755d7577-8fmcq\" (UID: \"463fa1da-5fff-4de1-815b-2f76070d5165\") " pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"
Jul 28 11:40:48.909830 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.909804    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/463fa1da-5fff-4de1-815b-2f76070d5165-webhook-cert\") pod \"opendatahub-operator-controller-manager-6755d7577-8fmcq\" (UID: \"463fa1da-5fff-4de1-815b-2f76070d5165\") " pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"
Jul 28 11:40:48.918917 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.918895    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6wwj\" (UniqueName: \"kubernetes.io/projected/463fa1da-5fff-4de1-815b-2f76070d5165-kube-api-access-f6wwj\") pod \"opendatahub-operator-controller-manager-6755d7577-8fmcq\" (UID: \"463fa1da-5fff-4de1-815b-2f76070d5165\") " pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"
Jul 28 11:40:48.983783 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:48.983721    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"
Jul 28 11:40:49.108697 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:49.108668    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"]
Jul 28 11:40:49.112045 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:40:49.112014    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463fa1da_5fff_4de1_815b_2f76070d5165.slice/crio-7fd292b0e39f9fbd25588fa18704b12b1b077a10196cc6bdfc5640fbfb53ab7e WatchSource:0}: Error finding container 7fd292b0e39f9fbd25588fa18704b12b1b077a10196cc6bdfc5640fbfb53ab7e: Status 404 returned error can't find the container with id 7fd292b0e39f9fbd25588fa18704b12b1b077a10196cc6bdfc5640fbfb53ab7e
Jul 28 11:40:49.514437 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:49.514336    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq" event={"ID":"463fa1da-5fff-4de1-815b-2f76070d5165","Type":"ContainerStarted","Data":"7fd292b0e39f9fbd25588fa18704b12b1b077a10196cc6bdfc5640fbfb53ab7e"}
Jul 28 11:40:52.527161 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:52.527092    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq" event={"ID":"463fa1da-5fff-4de1-815b-2f76070d5165","Type":"ContainerStarted","Data":"78d845cb31d578ac8585c699bc33ec97b8dc7218356a14da20cbce8513b3ecb5"}
Jul 28 11:40:53.532380 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:53.532300    2561 generic.go:356] "Generic (PLEG): container finished" podID="463fa1da-5fff-4de1-815b-2f76070d5165" containerID="78d845cb31d578ac8585c699bc33ec97b8dc7218356a14da20cbce8513b3ecb5" exitCode=0
Jul 28 11:40:53.532380 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:53.532361    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq" event={"ID":"463fa1da-5fff-4de1-815b-2f76070d5165","Type":"ContainerDied","Data":"78d845cb31d578ac8585c699bc33ec97b8dc7218356a14da20cbce8513b3ecb5"}
Jul 28 11:40:53.532847 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:53.532832    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 11:40:54.537596 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:54.537538    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq" event={"ID":"463fa1da-5fff-4de1-815b-2f76070d5165","Type":"ContainerStarted","Data":"590d61b99dc4a623a6a1eabc01490f388f561f9661568f18ded2c746533bb3ac"}
Jul 28 11:40:54.538018 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:54.537642    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"
Jul 28 11:40:54.561213 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:40:54.561162    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq" podStartSLOduration=2.161892247 podStartE2EDuration="6.561145685s" totalImagesPullingTime="3.326879382s" totalInitContainerRuntime="1.072374056s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:40:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:40:54.559487532 +0000 UTC m=+606.418143543" watchObservedRunningTime="2026-07-28 11:40:54.561145685 +0000 UTC m=+606.419801665"
Jul 28 11:41:05.542841 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:05.542812    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6755d7577-8fmcq"
Jul 28 11:41:17.087001 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.086927    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f"]
Jul 28 11:41:17.089959 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.089941    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f"
Jul 28 11:41:17.097446 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.097425    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 11:41:17.098077 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.098058    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 11:41:17.098169 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.098060    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 11:41:17.098169 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.098106    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 11:41:17.098252 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.098061    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cs4jt\""
Jul 28 11:41:17.098252 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.098098    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 11:41:17.108789 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.108768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f09f2213-fe3e-42f1-ad6f-e750825486da-cert\") pod \"lws-controller-manager-65bf6bb88c-rjm4f\" (UID: \"f09f2213-fe3e-42f1-ad6f-e750825486da\") " pod="openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f"
Jul 28 11:41:17.108944 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.108801    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg7fs\" (UniqueName: \"kubernetes.io/projected/f09f2213-fe3e-42f1-ad6f-e750825486da-kube-api-access-lg7fs\") pod \"lws-controller-manager-65bf6bb88c-rjm4f\" (UID: \"f09f2213-fe3e-42f1-ad6f-e750825486da\") " pod="openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f"
Jul 28 11:41:17.108944 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.108826    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f09f2213-fe3e-42f1-ad6f-e750825486da-manager-config\") pod \"lws-controller-manager-65bf6bb88c-rjm4f\" (UID: \"f09f2213-fe3e-42f1-ad6f-e750825486da\") " pod="openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f"
Jul 28 11:41:17.108944 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.108839    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f09f2213-fe3e-42f1-ad6f-e750825486da-metrics-cert\") pod \"lws-controller-manager-65bf6bb88c-rjm4f\" (UID: \"f09f2213-fe3e-42f1-ad6f-e750825486da\") " pod="openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f"
Jul 28 11:41:17.110464 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.110445    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f"]
Jul 28 11:41:17.209666 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.209640    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f09f2213-fe3e-42f1-ad6f-e750825486da-cert\") pod \"lws-controller-manager-65bf6bb88c-rjm4f\" (UID: \"f09f2213-fe3e-42f1-ad6f-e750825486da\") " pod="openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f"
Jul 28 11:41:17.209786 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.209676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg7fs\" (UniqueName: \"kubernetes.io/projected/f09f2213-fe3e-42f1-ad6f-e750825486da-kube-api-access-lg7fs\") pod \"lws-controller-manager-65bf6bb88c-rjm4f\" (UID: \"f09f2213-fe3e-42f1-ad6f-e750825486da\") " pod="openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f"
Jul 28 11:41:17.209786 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.209703    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f09f2213-fe3e-42f1-ad6f-e750825486da-manager-config\") pod \"lws-controller-manager-65bf6bb88c-rjm4f\" (UID: \"f09f2213-fe3e-42f1-ad6f-e750825486da\") " pod="openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f"
Jul 28 11:41:17.209786 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.209725    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f09f2213-fe3e-42f1-ad6f-e750825486da-metrics-cert\") pod \"lws-controller-manager-65bf6bb88c-rjm4f\" (UID: \"f09f2213-fe3e-42f1-ad6f-e750825486da\") " pod="openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f"
Jul 28 11:41:17.210305 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.210282    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f09f2213-fe3e-42f1-ad6f-e750825486da-manager-config\") pod \"lws-controller-manager-65bf6bb88c-rjm4f\" (UID: \"f09f2213-fe3e-42f1-ad6f-e750825486da\") " pod="openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f"
Jul 28 11:41:17.211973 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.211952    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f09f2213-fe3e-42f1-ad6f-e750825486da-metrics-cert\") pod \"lws-controller-manager-65bf6bb88c-rjm4f\" (UID: \"f09f2213-fe3e-42f1-ad6f-e750825486da\") " pod="openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f"
Jul 28 11:41:17.212052 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.211981    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f09f2213-fe3e-42f1-ad6f-e750825486da-cert\") pod \"lws-controller-manager-65bf6bb88c-rjm4f\" (UID: \"f09f2213-fe3e-42f1-ad6f-e750825486da\") " pod="openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f"
Jul 28 11:41:17.218853 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.218832    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg7fs\" (UniqueName: \"kubernetes.io/projected/f09f2213-fe3e-42f1-ad6f-e750825486da-kube-api-access-lg7fs\") pod \"lws-controller-manager-65bf6bb88c-rjm4f\" (UID: \"f09f2213-fe3e-42f1-ad6f-e750825486da\") " pod="openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f"
Jul 28 11:41:17.398743 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.398678    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f"
Jul 28 11:41:17.524038 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.524014    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f"]
Jul 28 11:41:17.527092 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:41:17.527068    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09f2213_fe3e_42f1_ad6f_e750825486da.slice/crio-a36dab3cce06e8251e4a9fbc66f5d71aa8253c341301b7912aa81ecc21aae756 WatchSource:0}: Error finding container a36dab3cce06e8251e4a9fbc66f5d71aa8253c341301b7912aa81ecc21aae756: Status 404 returned error can't find the container with id a36dab3cce06e8251e4a9fbc66f5d71aa8253c341301b7912aa81ecc21aae756
Jul 28 11:41:17.611529 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:17.611496    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f" event={"ID":"f09f2213-fe3e-42f1-ad6f-e750825486da","Type":"ContainerStarted","Data":"a36dab3cce06e8251e4a9fbc66f5d71aa8253c341301b7912aa81ecc21aae756"}
Jul 28 11:41:18.791358 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:18.791328    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-l6wtq"]
Jul 28 11:41:18.794749 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:18.794725    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l6wtq"
Jul 28 11:41:18.797521 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:18.797496    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 28 11:41:18.797637 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:18.797591    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-xhn8d\""
Jul 28 11:41:18.810219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:18.810198    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-l6wtq"]
Jul 28 11:41:18.820682 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:18.820657    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgwhs\" (UniqueName: \"kubernetes.io/projected/d7032b5a-b6d6-432a-9608-41553f3bf317-kube-api-access-lgwhs\") pod \"kserve-controller-manager-d6f4b65b8-l6wtq\" (UID: \"d7032b5a-b6d6-432a-9608-41553f3bf317\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-l6wtq"
Jul 28 11:41:18.820780 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:18.820703    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7032b5a-b6d6-432a-9608-41553f3bf317-cert\") pod \"kserve-controller-manager-d6f4b65b8-l6wtq\" (UID: \"d7032b5a-b6d6-432a-9608-41553f3bf317\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-l6wtq"
Jul 28 11:41:18.922025 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:18.921994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgwhs\" (UniqueName: \"kubernetes.io/projected/d7032b5a-b6d6-432a-9608-41553f3bf317-kube-api-access-lgwhs\") pod \"kserve-controller-manager-d6f4b65b8-l6wtq\" (UID: \"d7032b5a-b6d6-432a-9608-41553f3bf317\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-l6wtq"
Jul 28 11:41:18.922178 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:18.922039    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7032b5a-b6d6-432a-9608-41553f3bf317-cert\") pod \"kserve-controller-manager-d6f4b65b8-l6wtq\" (UID: \"d7032b5a-b6d6-432a-9608-41553f3bf317\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-l6wtq"
Jul 28 11:41:18.922178 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:41:18.922138    2561 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 28 11:41:18.922265 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:41:18.922191    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7032b5a-b6d6-432a-9608-41553f3bf317-cert podName:d7032b5a-b6d6-432a-9608-41553f3bf317 nodeName:}" failed. No retries permitted until 2026-07-28 11:41:19.422175511 +0000 UTC m=+631.280831490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d7032b5a-b6d6-432a-9608-41553f3bf317-cert") pod "kserve-controller-manager-d6f4b65b8-l6wtq" (UID: "d7032b5a-b6d6-432a-9608-41553f3bf317") : secret "kserve-webhook-server-cert" not found
Jul 28 11:41:18.937760 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:18.937737    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgwhs\" (UniqueName: \"kubernetes.io/projected/d7032b5a-b6d6-432a-9608-41553f3bf317-kube-api-access-lgwhs\") pod \"kserve-controller-manager-d6f4b65b8-l6wtq\" (UID: \"d7032b5a-b6d6-432a-9608-41553f3bf317\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-l6wtq"
Jul 28 11:41:19.427168 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:19.427129    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7032b5a-b6d6-432a-9608-41553f3bf317-cert\") pod \"kserve-controller-manager-d6f4b65b8-l6wtq\" (UID: \"d7032b5a-b6d6-432a-9608-41553f3bf317\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-l6wtq"
Jul 28 11:41:19.429990 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:19.429961    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7032b5a-b6d6-432a-9608-41553f3bf317-cert\") pod \"kserve-controller-manager-d6f4b65b8-l6wtq\" (UID: \"d7032b5a-b6d6-432a-9608-41553f3bf317\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-l6wtq"
Jul 28 11:41:19.707027 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:19.706941    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l6wtq"
Jul 28 11:41:20.073384 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:20.073284    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-l6wtq"]
Jul 28 11:41:20.075556 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:41:20.075528    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7032b5a_b6d6_432a_9608_41553f3bf317.slice/crio-91c3309d3339c671a0b7f2247c945e644dec147c8d7751909daf695a8ff06729 WatchSource:0}: Error finding container 91c3309d3339c671a0b7f2247c945e644dec147c8d7751909daf695a8ff06729: Status 404 returned error can't find the container with id 91c3309d3339c671a0b7f2247c945e644dec147c8d7751909daf695a8ff06729
Jul 28 11:41:20.624088 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:20.624053    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f" event={"ID":"f09f2213-fe3e-42f1-ad6f-e750825486da","Type":"ContainerStarted","Data":"f0858d2a72844cdd9a4dba502b414ef0c1bee0b29693d15c9f93815b4c7847c3"}
Jul 28 11:41:20.624268 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:20.624199    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f"
Jul 28 11:41:20.625288 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:20.625263    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l6wtq" event={"ID":"d7032b5a-b6d6-432a-9608-41553f3bf317","Type":"ContainerStarted","Data":"91c3309d3339c671a0b7f2247c945e644dec147c8d7751909daf695a8ff06729"}
Jul 28 11:41:20.645180 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:20.645141    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f" podStartSLOduration=1.211299973 podStartE2EDuration="3.645127455s" totalImagesPullingTime="2.433827482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:41:20.643535451 +0000 UTC m=+632.502191451" watchObservedRunningTime="2026-07-28 11:41:20.645127455 +0000 UTC m=+632.503783433"
Jul 28 11:41:22.634233 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:22.634205    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l6wtq" event={"ID":"d7032b5a-b6d6-432a-9608-41553f3bf317","Type":"ContainerStarted","Data":"c8898a57e0d98810fc433aaca92efdd7a85b1583a5a488b7b6d19435db9a017f"}
Jul 28 11:41:22.634529 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:22.634334    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l6wtq"
Jul 28 11:41:22.652141 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:22.652092    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l6wtq" podStartSLOduration=2.151077358 podStartE2EDuration="4.652072573s" totalImagesPullingTime="2.500995215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:41:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:41:22.650331944 +0000 UTC m=+634.508987949" watchObservedRunningTime="2026-07-28 11:41:22.652072573 +0000 UTC m=+634.510728574"
Jul 28 11:41:31.630986 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:31.630955    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-65bf6bb88c-rjm4f"
Jul 28 11:41:53.643123 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:41:53.643095    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-l6wtq"
Jul 28 11:42:51.236497 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:51.236416    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zwcm4"]
Jul 28 11:42:51.239526 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:51.239507    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zwcm4"
Jul 28 11:42:51.241567 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:51.241546    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 28 11:42:51.241723 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:51.241701    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 28 11:42:51.241799 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:51.241743    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 11:42:51.242182 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:51.242168    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-v6kx8\""
Jul 28 11:42:51.242252 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:51.242225    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 11:42:51.252482 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:51.252462    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zwcm4"]
Jul 28 11:42:51.321572 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:51.321551    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/16bdac86-0c58-4ceb-891c-a9b36d22e6be-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zwcm4\" (UID: \"16bdac86-0c58-4ceb-891c-a9b36d22e6be\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zwcm4"
Jul 28 11:42:51.321689 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:51.321619    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/16bdac86-0c58-4ceb-891c-a9b36d22e6be-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zwcm4\" (UID: \"16bdac86-0c58-4ceb-891c-a9b36d22e6be\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zwcm4"
Jul 28 11:42:51.321751 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:51.321696    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdlrw\" (UniqueName: \"kubernetes.io/projected/16bdac86-0c58-4ceb-891c-a9b36d22e6be-kube-api-access-wdlrw\") pod \"kuadrant-console-plugin-66b5774985-zwcm4\" (UID: \"16bdac86-0c58-4ceb-891c-a9b36d22e6be\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zwcm4"
Jul 28 11:42:51.422000 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:51.421981    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/16bdac86-0c58-4ceb-891c-a9b36d22e6be-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zwcm4\" (UID: \"16bdac86-0c58-4ceb-891c-a9b36d22e6be\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zwcm4"
Jul 28 11:42:51.422080 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:51.422034    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/16bdac86-0c58-4ceb-891c-a9b36d22e6be-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zwcm4\" (UID: \"16bdac86-0c58-4ceb-891c-a9b36d22e6be\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zwcm4"
Jul 28 11:42:51.422080 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:51.422062    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdlrw\" (UniqueName: \"kubernetes.io/projected/16bdac86-0c58-4ceb-891c-a9b36d22e6be-kube-api-access-wdlrw\") pod \"kuadrant-console-plugin-66b5774985-zwcm4\" (UID: \"16bdac86-0c58-4ceb-891c-a9b36d22e6be\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zwcm4"
Jul 28 11:42:51.422211 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:42:51.422194    2561 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 28 11:42:51.422275 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:42:51.422263    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16bdac86-0c58-4ceb-891c-a9b36d22e6be-plugin-serving-cert podName:16bdac86-0c58-4ceb-891c-a9b36d22e6be nodeName:}" failed. No retries permitted until 2026-07-28 11:42:51.922242191 +0000 UTC m=+723.780898173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/16bdac86-0c58-4ceb-891c-a9b36d22e6be-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-zwcm4" (UID: "16bdac86-0c58-4ceb-891c-a9b36d22e6be") : secret "plugin-serving-cert" not found
Jul 28 11:42:51.422564 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:51.422546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/16bdac86-0c58-4ceb-891c-a9b36d22e6be-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zwcm4\" (UID: \"16bdac86-0c58-4ceb-891c-a9b36d22e6be\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zwcm4"
Jul 28 11:42:51.436191 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:51.436170    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdlrw\" (UniqueName: \"kubernetes.io/projected/16bdac86-0c58-4ceb-891c-a9b36d22e6be-kube-api-access-wdlrw\") pod \"kuadrant-console-plugin-66b5774985-zwcm4\" (UID: \"16bdac86-0c58-4ceb-891c-a9b36d22e6be\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zwcm4"
Jul 28 11:42:51.924955 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:51.924919    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/16bdac86-0c58-4ceb-891c-a9b36d22e6be-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zwcm4\" (UID: \"16bdac86-0c58-4ceb-891c-a9b36d22e6be\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zwcm4"
Jul 28 11:42:51.927202 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:51.927170    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/16bdac86-0c58-4ceb-891c-a9b36d22e6be-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zwcm4\" (UID: \"16bdac86-0c58-4ceb-891c-a9b36d22e6be\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zwcm4"
Jul 28 11:42:52.148171 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:52.148142    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zwcm4"
Jul 28 11:42:52.275473 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:52.275450    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zwcm4"]
Jul 28 11:42:52.278017 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:42:52.277982    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16bdac86_0c58_4ceb_891c_a9b36d22e6be.slice/crio-1f8173fe8f7c69d07964df6db6942eba1ec43d1fb22b158b81a81d42064e8e9c WatchSource:0}: Error finding container 1f8173fe8f7c69d07964df6db6942eba1ec43d1fb22b158b81a81d42064e8e9c: Status 404 returned error can't find the container with id 1f8173fe8f7c69d07964df6db6942eba1ec43d1fb22b158b81a81d42064e8e9c
Jul 28 11:42:52.952142 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:42:52.952096    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zwcm4" event={"ID":"16bdac86-0c58-4ceb-891c-a9b36d22e6be","Type":"ContainerStarted","Data":"1f8173fe8f7c69d07964df6db6942eba1ec43d1fb22b158b81a81d42064e8e9c"}
Jul 28 11:43:03.047598 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:03.047552    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2"]
Jul 28 11:43:03.120415 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:03.120386    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2"]
Jul 28 11:43:03.120616 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:03.120517    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2"
Jul 28 11:43:03.122977 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:03.122954    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ljvfk\""
Jul 28 11:43:03.224931 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:03.224897    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2485872-d2ab-4070-96bf-ecc465548834-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mvlw2\" (UID: \"f2485872-d2ab-4070-96bf-ecc465548834\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2"
Jul 28 11:43:03.225088 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:03.224969    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzjh2\" (UniqueName: \"kubernetes.io/projected/f2485872-d2ab-4070-96bf-ecc465548834-kube-api-access-fzjh2\") pod \"kuadrant-operator-controller-manager-86878c769f-mvlw2\" (UID: \"f2485872-d2ab-4070-96bf-ecc465548834\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2"
Jul 28 11:43:03.325962 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:03.325927    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2485872-d2ab-4070-96bf-ecc465548834-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mvlw2\" (UID: \"f2485872-d2ab-4070-96bf-ecc465548834\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2"
Jul 28 11:43:03.326125 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:03.326013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzjh2\" (UniqueName: \"kubernetes.io/projected/f2485872-d2ab-4070-96bf-ecc465548834-kube-api-access-fzjh2\") pod \"kuadrant-operator-controller-manager-86878c769f-mvlw2\" (UID: \"f2485872-d2ab-4070-96bf-ecc465548834\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2"
Jul 28 11:43:03.326264 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:03.326160    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2485872-d2ab-4070-96bf-ecc465548834-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mvlw2\" (UID: \"f2485872-d2ab-4070-96bf-ecc465548834\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2"
Jul 28 11:43:03.345789 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:03.345764    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzjh2\" (UniqueName: \"kubernetes.io/projected/f2485872-d2ab-4070-96bf-ecc465548834-kube-api-access-fzjh2\") pod \"kuadrant-operator-controller-manager-86878c769f-mvlw2\" (UID: \"f2485872-d2ab-4070-96bf-ecc465548834\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2"
Jul 28 11:43:03.438326 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:03.438299    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2"
Jul 28 11:43:13.057891 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:13.057865    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2"]
Jul 28 11:43:13.061812 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:43:13.061783    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2485872_d2ab_4070_96bf_ecc465548834.slice/crio-5ad35c216f491b8769640c1c288b65f984e7ee927b3dbd38c757f456966ae706 WatchSource:0}: Error finding container 5ad35c216f491b8769640c1c288b65f984e7ee927b3dbd38c757f456966ae706: Status 404 returned error can't find the container with id 5ad35c216f491b8769640c1c288b65f984e7ee927b3dbd38c757f456966ae706
Jul 28 11:43:14.053272 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:14.053235    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2" event={"ID":"f2485872-d2ab-4070-96bf-ecc465548834","Type":"ContainerStarted","Data":"5ad35c216f491b8769640c1c288b65f984e7ee927b3dbd38c757f456966ae706"}
Jul 28 11:43:14.054526 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:14.054500    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zwcm4" event={"ID":"16bdac86-0c58-4ceb-891c-a9b36d22e6be","Type":"ContainerStarted","Data":"dc338ce47fa9388490795f6a3f6ccf2b610667eca86c929f500a6371aeff08f5"}
Jul 28 11:43:14.072109 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:14.072060    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zwcm4" podStartSLOduration=2.329537254 podStartE2EDuration="23.072045074s" totalImagesPullingTime="20.74250782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:43:14.070810106 +0000 UTC m=+745.929466106" watchObservedRunningTime="2026-07-28 11:43:14.072045074 +0000 UTC m=+745.930701073"
Jul 28 11:43:18.076378 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:18.076342    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2" event={"ID":"f2485872-d2ab-4070-96bf-ecc465548834","Type":"ContainerStarted","Data":"a56c38221764c88280513e4242d3a82fd8180eade9a823a110b5d894a2778365"}
Jul 28 11:43:18.076789 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:18.076489    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2"
Jul 28 11:43:18.096019 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:18.095976    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2" podStartSLOduration=10.413671526 podStartE2EDuration="15.095962879s" totalImagesPullingTime="4.682291353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:43:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:43:18.09508431 +0000 UTC m=+749.953740308" watchObservedRunningTime="2026-07-28 11:43:18.095962879 +0000 UTC m=+749.954618875"
Jul 28 11:43:29.082516 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:29.082487    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2"
Jul 28 11:43:51.799219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:51.799181    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:43:52.006767 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:52.006735    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:43:52.006767 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:52.006773    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:43:52.006952 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:52.006858    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"
Jul 28 11:43:52.009197 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:52.009168    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 28 11:43:52.101995 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:52.101969    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zpnc\" (UniqueName: \"kubernetes.io/projected/426d4fc8-ca99-4c4d-bef0-e2aab6ab7afa-kube-api-access-9zpnc\") pod \"limitador-limitador-54d6d988d5-6t8t4\" (UID: \"426d4fc8-ca99-4c4d-bef0-e2aab6ab7afa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"
Jul 28 11:43:52.102115 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:52.102005    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/426d4fc8-ca99-4c4d-bef0-e2aab6ab7afa-config-file\") pod \"limitador-limitador-54d6d988d5-6t8t4\" (UID: \"426d4fc8-ca99-4c4d-bef0-e2aab6ab7afa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"
Jul 28 11:43:52.202739 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:52.202714    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zpnc\" (UniqueName: \"kubernetes.io/projected/426d4fc8-ca99-4c4d-bef0-e2aab6ab7afa-kube-api-access-9zpnc\") pod \"limitador-limitador-54d6d988d5-6t8t4\" (UID: \"426d4fc8-ca99-4c4d-bef0-e2aab6ab7afa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"
Jul 28 11:43:52.202856 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:52.202751    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/426d4fc8-ca99-4c4d-bef0-e2aab6ab7afa-config-file\") pod \"limitador-limitador-54d6d988d5-6t8t4\" (UID: \"426d4fc8-ca99-4c4d-bef0-e2aab6ab7afa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"
Jul 28 11:43:52.203367 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:52.203348    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/426d4fc8-ca99-4c4d-bef0-e2aab6ab7afa-config-file\") pod \"limitador-limitador-54d6d988d5-6t8t4\" (UID: \"426d4fc8-ca99-4c4d-bef0-e2aab6ab7afa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"
Jul 28 11:43:52.219189 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:52.219162    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zpnc\" (UniqueName: \"kubernetes.io/projected/426d4fc8-ca99-4c4d-bef0-e2aab6ab7afa-kube-api-access-9zpnc\") pod \"limitador-limitador-54d6d988d5-6t8t4\" (UID: \"426d4fc8-ca99-4c4d-bef0-e2aab6ab7afa\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"
Jul 28 11:43:52.322384 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:52.322357    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"
Jul 28 11:43:52.455546 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:52.455512    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:43:52.458751 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:43:52.458720    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426d4fc8_ca99_4c4d_bef0_e2aab6ab7afa.slice/crio-51e4484f3600d89962f6bedcf8c74bb529c19f0e8bdf4efafb9a22a79f1b5fbf WatchSource:0}: Error finding container 51e4484f3600d89962f6bedcf8c74bb529c19f0e8bdf4efafb9a22a79f1b5fbf: Status 404 returned error can't find the container with id 51e4484f3600d89962f6bedcf8c74bb529c19f0e8bdf4efafb9a22a79f1b5fbf
Jul 28 11:43:53.214327 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:53.214241    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-6t8t4" event={"ID":"426d4fc8-ca99-4c4d-bef0-e2aab6ab7afa","Type":"ContainerStarted","Data":"51e4484f3600d89962f6bedcf8c74bb529c19f0e8bdf4efafb9a22a79f1b5fbf"}
Jul 28 11:43:55.223954 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:55.223868    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-6t8t4" event={"ID":"426d4fc8-ca99-4c4d-bef0-e2aab6ab7afa","Type":"ContainerStarted","Data":"b1562d631e88e3ef30b5f3b84c4f010aba5626db99913217a0f9f2a9d9318f91"}
Jul 28 11:43:55.224295 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:55.223974    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"
Jul 28 11:43:55.240458 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:43:55.240401    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-6t8t4" podStartSLOduration=1.7365505319999999 podStartE2EDuration="4.240383584s" totalImagesPullingTime="2.503833052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:43:55.240077916 +0000 UTC m=+787.098733916" watchObservedRunningTime="2026-07-28 11:43:55.240383584 +0000 UTC m=+787.099039611"
Jul 28 11:44:06.229602 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:44:06.229551    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"
Jul 28 11:45:06.259941 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:06.259907    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:45:22.955693 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:22.955659    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:45:28.723642 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:28.723607    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:45:35.900690 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:35.900658    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"]
Jul 28 11:45:35.903807 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:35.903790    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:35.905949 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:35.905923    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 28 11:45:35.906482 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:35.906464    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 28 11:45:35.906482 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:35.906471    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-n92jt\""
Jul 28 11:45:35.906654 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:35.906467    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 28 11:45:35.914928 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:35.914902    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"]
Jul 28 11:45:36.005219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.005192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca401824-eb63-4e42-a0b2-6d36572fc15d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4tm65\" (UID: \"ca401824-eb63-4e42-a0b2-6d36572fc15d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:36.005312 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.005227    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca401824-eb63-4e42-a0b2-6d36572fc15d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4tm65\" (UID: \"ca401824-eb63-4e42-a0b2-6d36572fc15d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:36.005312 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.005257    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca401824-eb63-4e42-a0b2-6d36572fc15d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4tm65\" (UID: \"ca401824-eb63-4e42-a0b2-6d36572fc15d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:36.005382 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.005299    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca401824-eb63-4e42-a0b2-6d36572fc15d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4tm65\" (UID: \"ca401824-eb63-4e42-a0b2-6d36572fc15d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:36.005415 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.005375    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx55w\" (UniqueName: \"kubernetes.io/projected/ca401824-eb63-4e42-a0b2-6d36572fc15d-kube-api-access-fx55w\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4tm65\" (UID: \"ca401824-eb63-4e42-a0b2-6d36572fc15d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:36.005447 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.005412    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca401824-eb63-4e42-a0b2-6d36572fc15d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4tm65\" (UID: \"ca401824-eb63-4e42-a0b2-6d36572fc15d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:36.106505 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.106478    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx55w\" (UniqueName: \"kubernetes.io/projected/ca401824-eb63-4e42-a0b2-6d36572fc15d-kube-api-access-fx55w\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4tm65\" (UID: \"ca401824-eb63-4e42-a0b2-6d36572fc15d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:36.106640 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.106522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca401824-eb63-4e42-a0b2-6d36572fc15d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4tm65\" (UID: \"ca401824-eb63-4e42-a0b2-6d36572fc15d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:36.106640 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.106566    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca401824-eb63-4e42-a0b2-6d36572fc15d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4tm65\" (UID: \"ca401824-eb63-4e42-a0b2-6d36572fc15d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:36.106640 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.106623    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca401824-eb63-4e42-a0b2-6d36572fc15d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4tm65\" (UID: \"ca401824-eb63-4e42-a0b2-6d36572fc15d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:36.106806 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.106653    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca401824-eb63-4e42-a0b2-6d36572fc15d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4tm65\" (UID: \"ca401824-eb63-4e42-a0b2-6d36572fc15d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:36.106806 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.106688    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca401824-eb63-4e42-a0b2-6d36572fc15d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4tm65\" (UID: \"ca401824-eb63-4e42-a0b2-6d36572fc15d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:36.107060 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.107012    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca401824-eb63-4e42-a0b2-6d36572fc15d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4tm65\" (UID: \"ca401824-eb63-4e42-a0b2-6d36572fc15d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:36.107060 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.107037    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca401824-eb63-4e42-a0b2-6d36572fc15d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4tm65\" (UID: \"ca401824-eb63-4e42-a0b2-6d36572fc15d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:36.107209 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.107095    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca401824-eb63-4e42-a0b2-6d36572fc15d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4tm65\" (UID: \"ca401824-eb63-4e42-a0b2-6d36572fc15d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:36.108910 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.108884    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca401824-eb63-4e42-a0b2-6d36572fc15d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4tm65\" (UID: \"ca401824-eb63-4e42-a0b2-6d36572fc15d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:36.109036 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.109019    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca401824-eb63-4e42-a0b2-6d36572fc15d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4tm65\" (UID: \"ca401824-eb63-4e42-a0b2-6d36572fc15d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:36.114131 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.114111    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx55w\" (UniqueName: \"kubernetes.io/projected/ca401824-eb63-4e42-a0b2-6d36572fc15d-kube-api-access-fx55w\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4tm65\" (UID: \"ca401824-eb63-4e42-a0b2-6d36572fc15d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:36.238992 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.238936    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:36.360958 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.360925    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"]
Jul 28 11:45:36.363826 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:45:36.363793    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca401824_eb63_4e42_a0b2_6d36572fc15d.slice/crio-8026f50ff530d1ca3a799f75b20b7300246f7ac9ccae37087fc94e1c509ffc4b WatchSource:0}: Error finding container 8026f50ff530d1ca3a799f75b20b7300246f7ac9ccae37087fc94e1c509ffc4b: Status 404 returned error can't find the container with id 8026f50ff530d1ca3a799f75b20b7300246f7ac9ccae37087fc94e1c509ffc4b
Jul 28 11:45:36.622857 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:36.622823    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65" event={"ID":"ca401824-eb63-4e42-a0b2-6d36572fc15d","Type":"ContainerStarted","Data":"8026f50ff530d1ca3a799f75b20b7300246f7ac9ccae37087fc94e1c509ffc4b"}
Jul 28 11:45:37.328546 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:37.328511    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:45:37.628531 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:37.628449    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65" event={"ID":"ca401824-eb63-4e42-a0b2-6d36572fc15d","Type":"ContainerStarted","Data":"6b65d2ebe7d3ddb13d176c74f2f7761b4211fd8346ac148ec67b571f5e115091"}
Jul 28 11:45:37.645160 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:37.645106    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65" podStartSLOduration=1.6045600580000001 podStartE2EDuration="2.645088311s" totalImagesPullingTime="1.040528253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:45:37.644154619 +0000 UTC m=+889.502810612" watchObservedRunningTime="2026-07-28 11:45:37.645088311 +0000 UTC m=+889.503744354"
Jul 28 11:45:45.152897 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:45.152858    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:45:46.239108 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:46.239069    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:46.239503 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:46.239140    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:46.251431 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:46.251410    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:45:46.677337 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:45:46.677307    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4tm65"
Jul 28 11:46:14.113201 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.113160    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"]
Jul 28 11:46:14.119872 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.119849    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:14.121892 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.121875    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 28 11:46:14.127809 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.127786    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"]
Jul 28 11:46:14.210265 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.210229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2b9f\" (UniqueName: \"kubernetes.io/projected/cda6b35e-45c0-40f9-996c-1bf409ba9a27-kube-api-access-m2b9f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j88rq\" (UID: \"cda6b35e-45c0-40f9-996c-1bf409ba9a27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:14.210424 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.210277    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cda6b35e-45c0-40f9-996c-1bf409ba9a27-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j88rq\" (UID: \"cda6b35e-45c0-40f9-996c-1bf409ba9a27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:14.210424 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.210312    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cda6b35e-45c0-40f9-996c-1bf409ba9a27-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j88rq\" (UID: \"cda6b35e-45c0-40f9-996c-1bf409ba9a27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:14.210424 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.210347    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cda6b35e-45c0-40f9-996c-1bf409ba9a27-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j88rq\" (UID: \"cda6b35e-45c0-40f9-996c-1bf409ba9a27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:14.210424 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.210369    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cda6b35e-45c0-40f9-996c-1bf409ba9a27-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j88rq\" (UID: \"cda6b35e-45c0-40f9-996c-1bf409ba9a27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:14.210424 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.210385    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cda6b35e-45c0-40f9-996c-1bf409ba9a27-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j88rq\" (UID: \"cda6b35e-45c0-40f9-996c-1bf409ba9a27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:14.311057 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.311027    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2b9f\" (UniqueName: \"kubernetes.io/projected/cda6b35e-45c0-40f9-996c-1bf409ba9a27-kube-api-access-m2b9f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j88rq\" (UID: \"cda6b35e-45c0-40f9-996c-1bf409ba9a27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:14.311200 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.311070    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cda6b35e-45c0-40f9-996c-1bf409ba9a27-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j88rq\" (UID: \"cda6b35e-45c0-40f9-996c-1bf409ba9a27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:14.311200 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.311104    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cda6b35e-45c0-40f9-996c-1bf409ba9a27-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j88rq\" (UID: \"cda6b35e-45c0-40f9-996c-1bf409ba9a27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:14.311200 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.311128    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cda6b35e-45c0-40f9-996c-1bf409ba9a27-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j88rq\" (UID: \"cda6b35e-45c0-40f9-996c-1bf409ba9a27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:14.311200 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.311149    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cda6b35e-45c0-40f9-996c-1bf409ba9a27-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j88rq\" (UID: \"cda6b35e-45c0-40f9-996c-1bf409ba9a27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:14.311381 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.311253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cda6b35e-45c0-40f9-996c-1bf409ba9a27-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j88rq\" (UID: \"cda6b35e-45c0-40f9-996c-1bf409ba9a27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:14.311480 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.311464    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cda6b35e-45c0-40f9-996c-1bf409ba9a27-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j88rq\" (UID: \"cda6b35e-45c0-40f9-996c-1bf409ba9a27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:14.311565 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.311528    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cda6b35e-45c0-40f9-996c-1bf409ba9a27-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j88rq\" (UID: \"cda6b35e-45c0-40f9-996c-1bf409ba9a27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:14.311746 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.311716    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cda6b35e-45c0-40f9-996c-1bf409ba9a27-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j88rq\" (UID: \"cda6b35e-45c0-40f9-996c-1bf409ba9a27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:14.313479 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.313460    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cda6b35e-45c0-40f9-996c-1bf409ba9a27-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j88rq\" (UID: \"cda6b35e-45c0-40f9-996c-1bf409ba9a27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:14.313782 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.313764    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cda6b35e-45c0-40f9-996c-1bf409ba9a27-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j88rq\" (UID: \"cda6b35e-45c0-40f9-996c-1bf409ba9a27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:14.320325 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.320304    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2b9f\" (UniqueName: \"kubernetes.io/projected/cda6b35e-45c0-40f9-996c-1bf409ba9a27-kube-api-access-m2b9f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j88rq\" (UID: \"cda6b35e-45c0-40f9-996c-1bf409ba9a27\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:14.431059 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.430980    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:14.560852 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.560822    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"]
Jul 28 11:46:14.563920 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:46:14.563888    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda6b35e_45c0_40f9_996c_1bf409ba9a27.slice/crio-ec8a87bf8cf28862612ce0956c6a0805986ae9817d4057c3fe7cbecce47d4bd1 WatchSource:0}: Error finding container ec8a87bf8cf28862612ce0956c6a0805986ae9817d4057c3fe7cbecce47d4bd1: Status 404 returned error can't find the container with id ec8a87bf8cf28862612ce0956c6a0805986ae9817d4057c3fe7cbecce47d4bd1
Jul 28 11:46:14.565018 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.564997    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 11:46:14.774260 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:14.774229    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq" event={"ID":"cda6b35e-45c0-40f9-996c-1bf409ba9a27","Type":"ContainerStarted","Data":"ec8a87bf8cf28862612ce0956c6a0805986ae9817d4057c3fe7cbecce47d4bd1"}
Jul 28 11:46:15.780423 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:15.780385    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq" event={"ID":"cda6b35e-45c0-40f9-996c-1bf409ba9a27","Type":"ContainerStarted","Data":"5402c152c97cda8df5a41e7e744763f692ed36ffa40553031de0030378fa10bc"}
Jul 28 11:46:15.799382 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:15.799336    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq" podStartSLOduration=1.607536072 podStartE2EDuration="1.799320134s" totalImagesPullingTime="191.784062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:46:15.799004718 +0000 UTC m=+927.657660717" watchObservedRunningTime="2026-07-28 11:46:15.799320134 +0000 UTC m=+927.657976132"
Jul 28 11:46:16.728568 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:16.728535    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:46:24.431125 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:24.431087    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:24.431624 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:24.431150    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:24.443419 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:24.443392    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:24.828476 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:24.828449    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j88rq"
Jul 28 11:46:34.534083 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:46:34.534046    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:48:26.872211 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:48:26.872171    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:48:38.321068 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:48:38.321031    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:48:46.824713 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:48:46.824632    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:50:30.334279 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:50:30.334241    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:50:41.422632 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:50:41.422592    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:51:19.323845 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:51:19.323811    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:51:26.840004 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:51:26.839971    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:51:37.425264 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:51:37.425227    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:51:45.735802 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:51:45.735725    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:51:54.845531 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:51:54.845495    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:52:07.039486 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:52:07.039451    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:52:17.531927 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:52:17.531892    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:52:26.734596 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:52:26.734547    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:52:51.839213 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:52:51.839179    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:53:04.026735 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:53:04.026700    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:53:16.926903 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:53:16.926823    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:53:24.838652 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:53:24.838618    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:53:41.324229 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:53:41.324195    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:53:51.420026 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:53:51.419991    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:54:07.733804 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:54:07.733768    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:54:17.427002 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:54:17.426967    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:54:27.533174 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:54:27.533141    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:54:39.023377 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:54:39.023337    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:54:47.823208 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:54:47.823128    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:55:15.038382 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:55:15.038347    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2"]
Jul 28 11:55:15.038806 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:55:15.038544    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2" podUID="f2485872-d2ab-4070-96bf-ecc465548834" containerName="manager" containerID="cri-o://a56c38221764c88280513e4242d3a82fd8180eade9a823a110b5d894a2778365" gracePeriod=10
Jul 28 11:55:16.583038 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:55:16.583016    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2"
Jul 28 11:55:16.723773 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:55:16.723693    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzjh2\" (UniqueName: \"kubernetes.io/projected/f2485872-d2ab-4070-96bf-ecc465548834-kube-api-access-fzjh2\") pod \"f2485872-d2ab-4070-96bf-ecc465548834\" (UID: \"f2485872-d2ab-4070-96bf-ecc465548834\") "
Jul 28 11:55:16.723773 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:55:16.723772    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2485872-d2ab-4070-96bf-ecc465548834-extensions-socket-volume\") pod \"f2485872-d2ab-4070-96bf-ecc465548834\" (UID: \"f2485872-d2ab-4070-96bf-ecc465548834\") "
Jul 28 11:55:16.724206 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:55:16.724178    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2485872-d2ab-4070-96bf-ecc465548834-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f2485872-d2ab-4070-96bf-ecc465548834" (UID: "f2485872-d2ab-4070-96bf-ecc465548834"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:55:16.725857 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:55:16.725835    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2485872-d2ab-4070-96bf-ecc465548834-kube-api-access-fzjh2" (OuterVolumeSpecName: "kube-api-access-fzjh2") pod "f2485872-d2ab-4070-96bf-ecc465548834" (UID: "f2485872-d2ab-4070-96bf-ecc465548834"). InnerVolumeSpecName "kube-api-access-fzjh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:55:16.824305 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:55:16.824277    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzjh2\" (UniqueName: \"kubernetes.io/projected/f2485872-d2ab-4070-96bf-ecc465548834-kube-api-access-fzjh2\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:55:16.824305 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:55:16.824299    2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2485872-d2ab-4070-96bf-ecc465548834-extensions-socket-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 11:55:16.973430 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:55:16.973398    2561 generic.go:356] "Generic (PLEG): container finished" podID="f2485872-d2ab-4070-96bf-ecc465548834" containerID="a56c38221764c88280513e4242d3a82fd8180eade9a823a110b5d894a2778365" exitCode=0
Jul 28 11:55:16.973606 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:55:16.973436    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2"
Jul 28 11:55:16.973606 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:55:16.973481    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2" event={"ID":"f2485872-d2ab-4070-96bf-ecc465548834","Type":"ContainerDied","Data":"a56c38221764c88280513e4242d3a82fd8180eade9a823a110b5d894a2778365"}
Jul 28 11:55:16.973606 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:55:16.973532    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2" event={"ID":"f2485872-d2ab-4070-96bf-ecc465548834","Type":"ContainerDied","Data":"5ad35c216f491b8769640c1c288b65f984e7ee927b3dbd38c757f456966ae706"}
Jul 28 11:55:16.973606 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:55:16.973554    2561 scope.go:117] "RemoveContainer" containerID="a56c38221764c88280513e4242d3a82fd8180eade9a823a110b5d894a2778365"
Jul 28 11:55:16.982880 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:55:16.982862    2561 scope.go:117] "RemoveContainer" containerID="a56c38221764c88280513e4242d3a82fd8180eade9a823a110b5d894a2778365"
Jul 28 11:55:16.983145 ip-10-0-141-155 kubenswrapper[2561]: E0728 11:55:16.983125    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56c38221764c88280513e4242d3a82fd8180eade9a823a110b5d894a2778365\": container with ID starting with a56c38221764c88280513e4242d3a82fd8180eade9a823a110b5d894a2778365 not found: ID does not exist" containerID="a56c38221764c88280513e4242d3a82fd8180eade9a823a110b5d894a2778365"
Jul 28 11:55:16.983219 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:55:16.983152    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56c38221764c88280513e4242d3a82fd8180eade9a823a110b5d894a2778365"} err="failed to get container status \"a56c38221764c88280513e4242d3a82fd8180eade9a823a110b5d894a2778365\": rpc error: code = NotFound desc = could not find container \"a56c38221764c88280513e4242d3a82fd8180eade9a823a110b5d894a2778365\": container with ID starting with a56c38221764c88280513e4242d3a82fd8180eade9a823a110b5d894a2778365 not found: ID does not exist"
Jul 28 11:55:16.994390 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:55:16.994368    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2"]
Jul 28 11:55:17.002081 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:55:17.002051    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvlw2"]
Jul 28 11:55:18.545907 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:55:18.545872    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2485872-d2ab-4070-96bf-ecc465548834" path="/var/lib/kubelet/pods/f2485872-d2ab-4070-96bf-ecc465548834/volumes"
Jul 28 11:56:16.380850 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:16.380818    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jpcq4"]
Jul 28 11:56:16.381317 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:16.381112    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2485872-d2ab-4070-96bf-ecc465548834" containerName="manager"
Jul 28 11:56:16.381317 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:16.381122    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2485872-d2ab-4070-96bf-ecc465548834" containerName="manager"
Jul 28 11:56:16.381317 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:16.381184    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2485872-d2ab-4070-96bf-ecc465548834" containerName="manager"
Jul 28 11:56:16.385455 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:16.385437    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jpcq4"
Jul 28 11:56:16.396645 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:16.396626    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n98tw\" (UniqueName: \"kubernetes.io/projected/9ea35308-e3df-4508-b643-50aa6aeb8ba8-kube-api-access-n98tw\") pod \"kuadrant-operator-controller-manager-86878c769f-jpcq4\" (UID: \"9ea35308-e3df-4508-b643-50aa6aeb8ba8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jpcq4"
Jul 28 11:56:16.396719 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:16.396668    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ea35308-e3df-4508-b643-50aa6aeb8ba8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jpcq4\" (UID: \"9ea35308-e3df-4508-b643-50aa6aeb8ba8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jpcq4"
Jul 28 11:56:16.397030 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:16.397015    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ljvfk\""
Jul 28 11:56:16.439830 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:16.439805    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jpcq4"]
Jul 28 11:56:16.497205 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:16.497171    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n98tw\" (UniqueName: \"kubernetes.io/projected/9ea35308-e3df-4508-b643-50aa6aeb8ba8-kube-api-access-n98tw\") pod \"kuadrant-operator-controller-manager-86878c769f-jpcq4\" (UID: \"9ea35308-e3df-4508-b643-50aa6aeb8ba8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jpcq4"
Jul 28 11:56:16.497327 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:16.497219    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ea35308-e3df-4508-b643-50aa6aeb8ba8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jpcq4\" (UID: \"9ea35308-e3df-4508-b643-50aa6aeb8ba8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jpcq4"
Jul 28 11:56:16.497559 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:16.497543    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ea35308-e3df-4508-b643-50aa6aeb8ba8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jpcq4\" (UID: \"9ea35308-e3df-4508-b643-50aa6aeb8ba8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jpcq4"
Jul 28 11:56:16.510210 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:16.510182    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n98tw\" (UniqueName: \"kubernetes.io/projected/9ea35308-e3df-4508-b643-50aa6aeb8ba8-kube-api-access-n98tw\") pod \"kuadrant-operator-controller-manager-86878c769f-jpcq4\" (UID: \"9ea35308-e3df-4508-b643-50aa6aeb8ba8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jpcq4"
Jul 28 11:56:16.694715 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:16.694641    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jpcq4"
Jul 28 11:56:16.824091 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:16.824066    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jpcq4"]
Jul 28 11:56:16.826896 ip-10-0-141-155 kubenswrapper[2561]: W0728 11:56:16.826871    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea35308_e3df_4508_b643_50aa6aeb8ba8.slice/crio-9b0ae37d83fb9c2804517d3d6d97164b438cce378b8828c0331b80f1806e785e WatchSource:0}: Error finding container 9b0ae37d83fb9c2804517d3d6d97164b438cce378b8828c0331b80f1806e785e: Status 404 returned error can't find the container with id 9b0ae37d83fb9c2804517d3d6d97164b438cce378b8828c0331b80f1806e785e
Jul 28 11:56:17.222988 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:17.222953    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jpcq4" event={"ID":"9ea35308-e3df-4508-b643-50aa6aeb8ba8","Type":"ContainerStarted","Data":"edef79fb73b0afa3dcf0943363ec1a15991f314c4966e6e25368271213824659"}
Jul 28 11:56:17.223154 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:17.223017    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jpcq4" event={"ID":"9ea35308-e3df-4508-b643-50aa6aeb8ba8","Type":"ContainerStarted","Data":"9b0ae37d83fb9c2804517d3d6d97164b438cce378b8828c0331b80f1806e785e"}
Jul 28 11:56:17.223154 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:17.223050    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jpcq4"
Jul 28 11:56:17.250388 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:17.250340    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jpcq4" podStartSLOduration=1.2503260840000001 podStartE2EDuration="1.250326084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:56:17.24777157 +0000 UTC m=+1529.106427569" watchObservedRunningTime="2026-07-28 11:56:17.250326084 +0000 UTC m=+1529.108982083"
Jul 28 11:56:28.229399 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:28.229362    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jpcq4"
Jul 28 11:56:36.333950 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:36.333914    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:56:46.824913 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:56:46.824878    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:57:04.527723 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:57:04.527689    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:57:14.426465 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:57:14.426431    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:57:26.931957 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:57:26.931925    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:57:34.433811 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:57:34.433772    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:57:44.824613 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:57:44.824518    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:57:52.539088 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:57:52.539056    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:58:02.231783 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:58:02.231754    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:58:11.132088 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:58:11.132054    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:58:21.938645 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:58:21.938608    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:58:32.345532 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:58:32.345494    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:58:54.334656 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:58:54.334620    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:59:04.237822 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:59:04.237785    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:59:14.638972 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:59:14.638900    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:59:27.622522 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:59:27.622484    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 11:59:51.033992 ip-10-0-141-155 kubenswrapper[2561]: I0728 11:59:51.033955    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:00:00.145573 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:00.145541    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754000-f6gf2"]
Jul 28 12:00:00.148779 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:00.148762    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754000-f6gf2"
Jul 28 12:00:00.150704 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:00.150680    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ts4x9\""
Jul 28 12:00:00.155792 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:00.155502    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754000-f6gf2"]
Jul 28 12:00:00.159425 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:00.159408    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754000-f6gf2"
Jul 28 12:00:00.285054 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:00.285022    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754000-f6gf2"]
Jul 28 12:00:00.288637 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:00:00.288611    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c02af9_91b9_48e9_8b4d_109366e7203a.slice/crio-91c15c8d0d8a5f84e95e2f0bc49c0aa23bee5789918e2c65f120212c6748ebe9 WatchSource:0}: Error finding container 91c15c8d0d8a5f84e95e2f0bc49c0aa23bee5789918e2c65f120212c6748ebe9: Status 404 returned error can't find the container with id 91c15c8d0d8a5f84e95e2f0bc49c0aa23bee5789918e2c65f120212c6748ebe9
Jul 28 12:00:00.290863 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:00.290847    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:00:01.113562 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:01.113524    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754000-f6gf2" event={"ID":"d3c02af9-91b9-48e9-8b4d-109366e7203a","Type":"ContainerStarted","Data":"91c15c8d0d8a5f84e95e2f0bc49c0aa23bee5789918e2c65f120212c6748ebe9"}
Jul 28 12:00:03.124217 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:03.124184    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754000-f6gf2" event={"ID":"d3c02af9-91b9-48e9-8b4d-109366e7203a","Type":"ContainerStarted","Data":"563eaa87e0513665f7e97517677637f9a3d7bea87e77caffb7875a62252725cc"}
Jul 28 12:00:03.142240 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:03.142140    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754000-f6gf2" podStartSLOduration=0.797144891 podStartE2EDuration="3.14212229s" totalImagesPullingTime="2.344977399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:00:03.140918965 +0000 UTC m=+1754.999574977" watchObservedRunningTime="2026-07-28 12:00:03.14212229 +0000 UTC m=+1755.000778289"
Jul 28 12:00:04.129879 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:04.129844    2561 generic.go:356] "Generic (PLEG): container finished" podID="d3c02af9-91b9-48e9-8b4d-109366e7203a" containerID="563eaa87e0513665f7e97517677637f9a3d7bea87e77caffb7875a62252725cc" exitCode=0
Jul 28 12:00:04.130294 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:04.129929    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754000-f6gf2" event={"ID":"d3c02af9-91b9-48e9-8b4d-109366e7203a","Type":"ContainerDied","Data":"563eaa87e0513665f7e97517677637f9a3d7bea87e77caffb7875a62252725cc"}
Jul 28 12:00:05.226219 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:05.226188    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:00:05.265874 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:05.265847    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754000-f6gf2"
Jul 28 12:00:06.139348 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:06.139316    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754000-f6gf2"
Jul 28 12:00:06.139348 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:06.139324    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754000-f6gf2" event={"ID":"d3c02af9-91b9-48e9-8b4d-109366e7203a","Type":"ContainerDied","Data":"91c15c8d0d8a5f84e95e2f0bc49c0aa23bee5789918e2c65f120212c6748ebe9"}
Jul 28 12:00:06.139689 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:06.139376    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91c15c8d0d8a5f84e95e2f0bc49c0aa23bee5789918e2c65f120212c6748ebe9"
Jul 28 12:00:12.329048 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:12.329018    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:00:29.348031 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:29.347994    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:00:38.353799 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:38.353770    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:00:47.738159 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:47.738130    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:00:58.443929 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:00:58.443892    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:01:07.940254 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:01:07.940222    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:01:17.047104 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:01:17.047061    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:01:30.028487 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:01:30.028456    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:01:39.427278 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:01:39.427222    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:01:48.733275 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:01:48.733243    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:02:00.637041 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:02:00.637005    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:02:12.427114 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:02:12.427038    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:02:21.133944 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:02:21.133905    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:02:32.530816 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:02:32.530779    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:02:43.527950 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:02:43.527915    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:02:52.332874 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:02:52.332841    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:03:00.232433 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:00.232388    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:03:08.826667 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:08.826633    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:03:23.029369 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:23.029336    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:03:32.140057 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:32.140024    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:03:45.031217 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:45.031143    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg"]
Jul 28 12:03:45.031591 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:45.031437    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c02af9-91b9-48e9-8b4d-109366e7203a" containerName="cleanup"
Jul 28 12:03:45.031591 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:45.031448    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c02af9-91b9-48e9-8b4d-109366e7203a" containerName="cleanup"
Jul 28 12:03:45.031591 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:45.031515    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c02af9-91b9-48e9-8b4d-109366e7203a" containerName="cleanup"
Jul 28 12:03:45.034649 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:45.034632    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg"
Jul 28 12:03:45.037450 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:45.037424    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 12:03:45.037450 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:45.037427    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7e937708\""
Jul 28 12:03:45.038004 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:45.037989    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 12:03:45.038066 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:45.038001    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7e937708-dockercfg-rbk6n\""
Jul 28 12:03:45.045848 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:45.045826    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg"]
Jul 28 12:03:45.125809 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:45.125770    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7-plugins-config-volume\") pod \"payload-processing-e2e-ait-7e937708-5db85cbd4-twshg\" (UID: \"17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7\") " pod="openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg"
Jul 28 12:03:45.125993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:45.125885    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fwf8\" (UniqueName: \"kubernetes.io/projected/17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7-kube-api-access-2fwf8\") pod \"payload-processing-e2e-ait-7e937708-5db85cbd4-twshg\" (UID: \"17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7\") " pod="openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg"
Jul 28 12:03:45.227278 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:45.227246    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fwf8\" (UniqueName: \"kubernetes.io/projected/17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7-kube-api-access-2fwf8\") pod \"payload-processing-e2e-ait-7e937708-5db85cbd4-twshg\" (UID: \"17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7\") " pod="openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg"
Jul 28 12:03:45.227448 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:45.227312    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7-plugins-config-volume\") pod \"payload-processing-e2e-ait-7e937708-5db85cbd4-twshg\" (UID: \"17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7\") " pod="openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg"
Jul 28 12:03:45.227991 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:45.227971    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7-plugins-config-volume\") pod \"payload-processing-e2e-ait-7e937708-5db85cbd4-twshg\" (UID: \"17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7\") " pod="openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg"
Jul 28 12:03:45.235471 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:45.235442    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fwf8\" (UniqueName: \"kubernetes.io/projected/17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7-kube-api-access-2fwf8\") pod \"payload-processing-e2e-ait-7e937708-5db85cbd4-twshg\" (UID: \"17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7\") " pod="openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg"
Jul 28 12:03:45.344607 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:45.344565    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg"
Jul 28 12:03:45.475672 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:45.475647    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg"]
Jul 28 12:03:45.478305 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:03:45.478279    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ecfa51_dbd7_443d_9ea1_dcea0ba38ed7.slice/crio-f411b6e60e6b660f914c3bef65ebc17ff4eceb527b540078d3da20f9566ac65c WatchSource:0}: Error finding container f411b6e60e6b660f914c3bef65ebc17ff4eceb527b540078d3da20f9566ac65c: Status 404 returned error can't find the container with id f411b6e60e6b660f914c3bef65ebc17ff4eceb527b540078d3da20f9566ac65c
Jul 28 12:03:46.035167 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:46.035134    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg" event={"ID":"17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7","Type":"ContainerStarted","Data":"f411b6e60e6b660f914c3bef65ebc17ff4eceb527b540078d3da20f9566ac65c"}
Jul 28 12:03:48.045905 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:48.045816    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg" event={"ID":"17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7","Type":"ContainerStarted","Data":"fcbcf81c5510fb7a56216f2cf3d275d18ef375ab70ce0bb9df8441a16e7e274c"}
Jul 28 12:03:48.046277 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:48.045964    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg"
Jul 28 12:03:48.062844 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:48.062793    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg" podStartSLOduration=0.774832497 podStartE2EDuration="3.062780558s" totalImagesPullingTime="2.287948061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:03:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:03:48.061135401 +0000 UTC m=+1979.919791400" watchObservedRunningTime="2026-07-28 12:03:48.062780558 +0000 UTC m=+1979.921436557"
Jul 28 12:03:48.430319 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:48.430286    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:03:59.052247 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:03:59.052218    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg"
Jul 28 12:04:09.672661 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:09.672626    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e937708-68h26"]
Jul 28 12:04:09.676117 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:09.676098    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e937708-68h26"
Jul 28 12:04:09.678105 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:09.678082    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 12:04:09.678331 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:09.678314    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ts4x9\""
Jul 28 12:04:09.683557 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:09.683274    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e937708-68h26"]
Jul 28 12:04:09.730658 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:09.730618    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa43120-dc2f-460b-94f2-d61c0f613f42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e937708-68h26\" (UID: \"bfa43120-dc2f-460b-94f2-d61c0f613f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e937708-68h26"
Jul 28 12:04:09.831910 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:09.831866    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa43120-dc2f-460b-94f2-d61c0f613f42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e937708-68h26\" (UID: \"bfa43120-dc2f-460b-94f2-d61c0f613f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e937708-68h26"
Jul 28 12:04:09.832487 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:09.832467    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa43120-dc2f-460b-94f2-d61c0f613f42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e937708-68h26\" (UID: \"bfa43120-dc2f-460b-94f2-d61c0f613f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e937708-68h26"
Jul 28 12:04:09.987154 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:09.987083    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e937708-68h26"
Jul 28 12:04:10.111831 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:10.111804    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e937708-68h26"]
Jul 28 12:04:10.114192 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:04:10.114162    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa43120_dc2f_460b_94f2_d61c0f613f42.slice/crio-42202d31ef013c31425ad6dbbdcf2e35a13ae6a56f1d3ee5a9c5a0f062cf3164 WatchSource:0}: Error finding container 42202d31ef013c31425ad6dbbdcf2e35a13ae6a56f1d3ee5a9c5a0f062cf3164: Status 404 returned error can't find the container with id 42202d31ef013c31425ad6dbbdcf2e35a13ae6a56f1d3ee5a9c5a0f062cf3164
Jul 28 12:04:10.138201 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:10.138176    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e937708-68h26" event={"ID":"bfa43120-dc2f-460b-94f2-d61c0f613f42","Type":"ContainerStarted","Data":"42202d31ef013c31425ad6dbbdcf2e35a13ae6a56f1d3ee5a9c5a0f062cf3164"}
Jul 28 12:04:11.143251 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:11.143214    2561 generic.go:356] "Generic (PLEG): container finished" podID="bfa43120-dc2f-460b-94f2-d61c0f613f42" containerID="fd04e82b5b738e580b135a234166d918f5d74f4bca44eba099b02e559f55fc74" exitCode=0
Jul 28 12:04:11.143647 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:11.143279    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e937708-68h26" event={"ID":"bfa43120-dc2f-460b-94f2-d61c0f613f42","Type":"ContainerDied","Data":"fd04e82b5b738e580b135a234166d918f5d74f4bca44eba099b02e559f55fc74"}
Jul 28 12:04:12.268379 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:12.268353    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e937708-68h26"
Jul 28 12:04:12.351228 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:12.351203    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa43120-dc2f-460b-94f2-d61c0f613f42-maas-api-service-ca\") pod \"bfa43120-dc2f-460b-94f2-d61c0f613f42\" (UID: \"bfa43120-dc2f-460b-94f2-d61c0f613f42\") "
Jul 28 12:04:12.351552 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:12.351532    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa43120-dc2f-460b-94f2-d61c0f613f42-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bfa43120-dc2f-460b-94f2-d61c0f613f42" (UID: "bfa43120-dc2f-460b-94f2-d61c0f613f42"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:04:12.452497 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:12.452432    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa43120-dc2f-460b-94f2-d61c0f613f42-maas-api-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:04:13.152266 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:13.152236    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e937708-68h26"
Jul 28 12:04:13.152266 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:13.152250    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e937708-68h26" event={"ID":"bfa43120-dc2f-460b-94f2-d61c0f613f42","Type":"ContainerDied","Data":"42202d31ef013c31425ad6dbbdcf2e35a13ae6a56f1d3ee5a9c5a0f062cf3164"}
Jul 28 12:04:13.152456 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:13.152290    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42202d31ef013c31425ad6dbbdcf2e35a13ae6a56f1d3ee5a9c5a0f062cf3164"
Jul 28 12:04:20.393416 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:20.393380    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg"]
Jul 28 12:04:20.393859 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:20.393598    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg" podUID="17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7" containerName="payload-processing" containerID="cri-o://fcbcf81c5510fb7a56216f2cf3d275d18ef375ab70ce0bb9df8441a16e7e274c" gracePeriod=30
Jul 28 12:04:20.645350 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:20.645292    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg"
Jul 28 12:04:20.714199 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:20.714165    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7-plugins-config-volume\") pod \"17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7\" (UID: \"17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7\") "
Jul 28 12:04:20.714365 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:20.714226    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fwf8\" (UniqueName: \"kubernetes.io/projected/17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7-kube-api-access-2fwf8\") pod \"17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7\" (UID: \"17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7\") "
Jul 28 12:04:20.714511 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:20.714488    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7" (UID: "17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:04:20.716421 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:20.716394    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7-kube-api-access-2fwf8" (OuterVolumeSpecName: "kube-api-access-2fwf8") pod "17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7" (UID: "17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7"). InnerVolumeSpecName "kube-api-access-2fwf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:04:20.815897 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:20.815863    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:04:20.815897 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:20.815892    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fwf8\" (UniqueName: \"kubernetes.io/projected/17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7-kube-api-access-2fwf8\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:04:21.186262 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:21.186227    2561 generic.go:356] "Generic (PLEG): container finished" podID="17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7" containerID="fcbcf81c5510fb7a56216f2cf3d275d18ef375ab70ce0bb9df8441a16e7e274c" exitCode=0
Jul 28 12:04:21.186416 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:21.186268    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg"
Jul 28 12:04:21.186416 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:21.186306    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg" event={"ID":"17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7","Type":"ContainerDied","Data":"fcbcf81c5510fb7a56216f2cf3d275d18ef375ab70ce0bb9df8441a16e7e274c"}
Jul 28 12:04:21.186416 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:21.186358    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg" event={"ID":"17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7","Type":"ContainerDied","Data":"f411b6e60e6b660f914c3bef65ebc17ff4eceb527b540078d3da20f9566ac65c"}
Jul 28 12:04:21.186416 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:21.186380    2561 scope.go:117] "RemoveContainer" containerID="fcbcf81c5510fb7a56216f2cf3d275d18ef375ab70ce0bb9df8441a16e7e274c"
Jul 28 12:04:21.194888 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:21.194871    2561 scope.go:117] "RemoveContainer" containerID="fcbcf81c5510fb7a56216f2cf3d275d18ef375ab70ce0bb9df8441a16e7e274c"
Jul 28 12:04:21.195139 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:04:21.195121    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcbcf81c5510fb7a56216f2cf3d275d18ef375ab70ce0bb9df8441a16e7e274c\": container with ID starting with fcbcf81c5510fb7a56216f2cf3d275d18ef375ab70ce0bb9df8441a16e7e274c not found: ID does not exist" containerID="fcbcf81c5510fb7a56216f2cf3d275d18ef375ab70ce0bb9df8441a16e7e274c"
Jul 28 12:04:21.195183 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:21.195149    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcbcf81c5510fb7a56216f2cf3d275d18ef375ab70ce0bb9df8441a16e7e274c"} err="failed to get container status \"fcbcf81c5510fb7a56216f2cf3d275d18ef375ab70ce0bb9df8441a16e7e274c\": rpc error: code = NotFound desc = could not find container \"fcbcf81c5510fb7a56216f2cf3d275d18ef375ab70ce0bb9df8441a16e7e274c\": container with ID starting with fcbcf81c5510fb7a56216f2cf3d275d18ef375ab70ce0bb9df8441a16e7e274c not found: ID does not exist"
Jul 28 12:04:21.206156 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:21.206131    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg"]
Jul 28 12:04:21.208032 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:21.208012    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e937708-5db85cbd4-twshg"]
Jul 28 12:04:22.546386 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:22.546350    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7" path="/var/lib/kubelet/pods/17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7/volumes"
Jul 28 12:04:23.241595 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:23.241548    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:04:24.841622 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:24.841575    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e937708-68h26"]
Jul 28 12:04:24.846951 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:24.846926    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e937708-68h26"]
Jul 28 12:04:26.546027 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:26.545994    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfa43120-dc2f-460b-94f2-d61c0f613f42" path="/var/lib/kubelet/pods/bfa43120-dc2f-460b-94f2-d61c0f613f42/volumes"
Jul 28 12:04:30.432096 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.432061    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9"]
Jul 28 12:04:30.432473 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.432343    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfa43120-dc2f-460b-94f2-d61c0f613f42" containerName="revoke-keys"
Jul 28 12:04:30.432473 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.432354    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfa43120-dc2f-460b-94f2-d61c0f613f42" containerName="revoke-keys"
Jul 28 12:04:30.432473 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.432363    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7" containerName="payload-processing"
Jul 28 12:04:30.432473 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.432368    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7" containerName="payload-processing"
Jul 28 12:04:30.432473 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.432429    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="17ecfa51-dbd7-443d-9ea1-dcea0ba38ed7" containerName="payload-processing"
Jul 28 12:04:30.432473 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.432440    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfa43120-dc2f-460b-94f2-d61c0f613f42" containerName="revoke-keys"
Jul 28 12:04:30.435531 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.435515    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9"
Jul 28 12:04:30.438911 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.438885    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c98bc341-dockercfg-6lt4h\""
Jul 28 12:04:30.439033 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.438919    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 12:04:30.439033 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.438979    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c98bc341\""
Jul 28 12:04:30.439033 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.439023    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 12:04:30.475965 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.475940    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9"]
Jul 28 12:04:30.497822 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.497796    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v97dp\" (UniqueName: \"kubernetes.io/projected/9cad37a3-e6bf-4b7e-bb95-4ff676613c51-kube-api-access-v97dp\") pod \"payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9\" (UID: \"9cad37a3-e6bf-4b7e-bb95-4ff676613c51\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9"
Jul 28 12:04:30.497925 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.497841    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cad37a3-e6bf-4b7e-bb95-4ff676613c51-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9\" (UID: \"9cad37a3-e6bf-4b7e-bb95-4ff676613c51\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9"
Jul 28 12:04:30.511195 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.511175    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn"]
Jul 28 12:04:30.514222 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.514206    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn"
Jul 28 12:04:30.549163 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.549135    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn"]
Jul 28 12:04:30.598431 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.598404    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v97dp\" (UniqueName: \"kubernetes.io/projected/9cad37a3-e6bf-4b7e-bb95-4ff676613c51-kube-api-access-v97dp\") pod \"payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9\" (UID: \"9cad37a3-e6bf-4b7e-bb95-4ff676613c51\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9"
Jul 28 12:04:30.598575 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.598447    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f93340a-0884-496b-9a37-c9406e7ffccf-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn\" (UID: \"4f93340a-0884-496b-9a37-c9406e7ffccf\") " pod="openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn"
Jul 28 12:04:30.598575 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.598485    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cad37a3-e6bf-4b7e-bb95-4ff676613c51-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9\" (UID: \"9cad37a3-e6bf-4b7e-bb95-4ff676613c51\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9"
Jul 28 12:04:30.598575 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.598534    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f8vz\" (UniqueName: \"kubernetes.io/projected/4f93340a-0884-496b-9a37-c9406e7ffccf-kube-api-access-4f8vz\") pod \"payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn\" (UID: \"4f93340a-0884-496b-9a37-c9406e7ffccf\") " pod="openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn"
Jul 28 12:04:30.599233 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.599213    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cad37a3-e6bf-4b7e-bb95-4ff676613c51-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9\" (UID: \"9cad37a3-e6bf-4b7e-bb95-4ff676613c51\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9"
Jul 28 12:04:30.612503 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.612475    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v97dp\" (UniqueName: \"kubernetes.io/projected/9cad37a3-e6bf-4b7e-bb95-4ff676613c51-kube-api-access-v97dp\") pod \"payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9\" (UID: \"9cad37a3-e6bf-4b7e-bb95-4ff676613c51\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9"
Jul 28 12:04:30.699907 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.699835    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f93340a-0884-496b-9a37-c9406e7ffccf-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn\" (UID: \"4f93340a-0884-496b-9a37-c9406e7ffccf\") " pod="openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn"
Jul 28 12:04:30.699907 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.699885    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f8vz\" (UniqueName: \"kubernetes.io/projected/4f93340a-0884-496b-9a37-c9406e7ffccf-kube-api-access-4f8vz\") pod \"payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn\" (UID: \"4f93340a-0884-496b-9a37-c9406e7ffccf\") " pod="openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn"
Jul 28 12:04:30.700545 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.700519    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f93340a-0884-496b-9a37-c9406e7ffccf-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn\" (UID: \"4f93340a-0884-496b-9a37-c9406e7ffccf\") " pod="openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn"
Jul 28 12:04:30.708339 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.708318    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f8vz\" (UniqueName: \"kubernetes.io/projected/4f93340a-0884-496b-9a37-c9406e7ffccf-kube-api-access-4f8vz\") pod \"payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn\" (UID: \"4f93340a-0884-496b-9a37-c9406e7ffccf\") " pod="openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn"
Jul 28 12:04:30.745654 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.745636    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9"
Jul 28 12:04:30.822927 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.822897    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn"
Jul 28 12:04:30.877527 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.876911    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9"]
Jul 28 12:04:30.880219 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:04:30.880186    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cad37a3_e6bf_4b7e_bb95_4ff676613c51.slice/crio-94e9ae76feadb7100e6923ff945bf889201edf0de7a91e9ef70fbe62807f2361 WatchSource:0}: Error finding container 94e9ae76feadb7100e6923ff945bf889201edf0de7a91e9ef70fbe62807f2361: Status 404 returned error can't find the container with id 94e9ae76feadb7100e6923ff945bf889201edf0de7a91e9ef70fbe62807f2361
Jul 28 12:04:30.974974 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:30.974942    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn"]
Jul 28 12:04:30.980480 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:04:30.980457    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f93340a_0884_496b_9a37_c9406e7ffccf.slice/crio-1c83d0c73263d7659ebfc4a82841a583eb02a15b2cda150202fdc0069e82e3a4 WatchSource:0}: Error finding container 1c83d0c73263d7659ebfc4a82841a583eb02a15b2cda150202fdc0069e82e3a4: Status 404 returned error can't find the container with id 1c83d0c73263d7659ebfc4a82841a583eb02a15b2cda150202fdc0069e82e3a4
Jul 28 12:04:31.228182 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:31.228096    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9" event={"ID":"9cad37a3-e6bf-4b7e-bb95-4ff676613c51","Type":"ContainerStarted","Data":"60bd80ceb2ec74cd851e6fc3490a9215d3f7b3a7ad071a4ae23d31f18ee3e83d"}
Jul 28 12:04:31.228335 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:31.228252    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9"
Jul 28 12:04:31.228335 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:31.228307    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9" event={"ID":"9cad37a3-e6bf-4b7e-bb95-4ff676613c51","Type":"ContainerStarted","Data":"94e9ae76feadb7100e6923ff945bf889201edf0de7a91e9ef70fbe62807f2361"}
Jul 28 12:04:31.229675 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:31.229654    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn" event={"ID":"4f93340a-0884-496b-9a37-c9406e7ffccf","Type":"ContainerStarted","Data":"082ea7712e8defd76195ba83ce513c2937756ce428d983c656c730af88a72a0f"}
Jul 28 12:04:31.229769 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:31.229688    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn" event={"ID":"4f93340a-0884-496b-9a37-c9406e7ffccf","Type":"ContainerStarted","Data":"1c83d0c73263d7659ebfc4a82841a583eb02a15b2cda150202fdc0069e82e3a4"}
Jul 28 12:04:31.229815 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:31.229798    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn"
Jul 28 12:04:31.244704 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:31.244662    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9" podStartSLOduration=1.2446503309999999 podStartE2EDuration="1.244650331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:31.243045722 +0000 UTC m=+2023.101701720" watchObservedRunningTime="2026-07-28 12:04:31.244650331 +0000 UTC m=+2023.103306330"
Jul 28 12:04:31.260854 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:31.260815    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn" podStartSLOduration=1.260801395 podStartE2EDuration="1.260801395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:04:31.260667512 +0000 UTC m=+2023.119323513" watchObservedRunningTime="2026-07-28 12:04:31.260801395 +0000 UTC m=+2023.119457395"
Jul 28 12:04:31.934459 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:31.934423    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:04:42.236595 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:42.236547    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn"
Jul 28 12:04:42.237034 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:04:42.236723    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9"
Jul 28 12:05:05.574813 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:05.574783    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn"]
Jul 28 12:05:05.575275 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:05.575056    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn" podUID="4f93340a-0884-496b-9a37-c9406e7ffccf" containerName="payload-processing" containerID="cri-o://082ea7712e8defd76195ba83ce513c2937756ce428d983c656c730af88a72a0f" gracePeriod=30
Jul 28 12:05:05.594611 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:05.594559    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9"]
Jul 28 12:05:05.594833 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:05.594792    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9" podUID="9cad37a3-e6bf-4b7e-bb95-4ff676613c51" containerName="payload-pre-processing" containerID="cri-o://60bd80ceb2ec74cd851e6fc3490a9215d3f7b3a7ad071a4ae23d31f18ee3e83d" gracePeriod=30
Jul 28 12:05:05.863797 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:05.863772    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn"
Jul 28 12:05:05.866655 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:05.866637    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9"
Jul 28 12:05:05.888565 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:05.888228    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f93340a-0884-496b-9a37-c9406e7ffccf-plugins-config-volume\") pod \"4f93340a-0884-496b-9a37-c9406e7ffccf\" (UID: \"4f93340a-0884-496b-9a37-c9406e7ffccf\") "
Jul 28 12:05:05.888565 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:05.888269    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f8vz\" (UniqueName: \"kubernetes.io/projected/4f93340a-0884-496b-9a37-c9406e7ffccf-kube-api-access-4f8vz\") pod \"4f93340a-0884-496b-9a37-c9406e7ffccf\" (UID: \"4f93340a-0884-496b-9a37-c9406e7ffccf\") "
Jul 28 12:05:05.888565 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:05.888327    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cad37a3-e6bf-4b7e-bb95-4ff676613c51-plugins-config-volume\") pod \"9cad37a3-e6bf-4b7e-bb95-4ff676613c51\" (UID: \"9cad37a3-e6bf-4b7e-bb95-4ff676613c51\") "
Jul 28 12:05:05.888565 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:05.888393    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v97dp\" (UniqueName: \"kubernetes.io/projected/9cad37a3-e6bf-4b7e-bb95-4ff676613c51-kube-api-access-v97dp\") pod \"9cad37a3-e6bf-4b7e-bb95-4ff676613c51\" (UID: \"9cad37a3-e6bf-4b7e-bb95-4ff676613c51\") "
Jul 28 12:05:05.889062 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:05.889020    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cad37a3-e6bf-4b7e-bb95-4ff676613c51-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9cad37a3-e6bf-4b7e-bb95-4ff676613c51" (UID: "9cad37a3-e6bf-4b7e-bb95-4ff676613c51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:05:05.889660 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:05.889383    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f93340a-0884-496b-9a37-c9406e7ffccf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f93340a-0884-496b-9a37-c9406e7ffccf" (UID: "4f93340a-0884-496b-9a37-c9406e7ffccf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:05:05.891479 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:05.891454    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cad37a3-e6bf-4b7e-bb95-4ff676613c51-kube-api-access-v97dp" (OuterVolumeSpecName: "kube-api-access-v97dp") pod "9cad37a3-e6bf-4b7e-bb95-4ff676613c51" (UID: "9cad37a3-e6bf-4b7e-bb95-4ff676613c51"). InnerVolumeSpecName "kube-api-access-v97dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:05:05.892137 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:05.892114    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f93340a-0884-496b-9a37-c9406e7ffccf-kube-api-access-4f8vz" (OuterVolumeSpecName: "kube-api-access-4f8vz") pod "4f93340a-0884-496b-9a37-c9406e7ffccf" (UID: "4f93340a-0884-496b-9a37-c9406e7ffccf"). InnerVolumeSpecName "kube-api-access-4f8vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:05:05.989812 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:05.989768    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v97dp\" (UniqueName: \"kubernetes.io/projected/9cad37a3-e6bf-4b7e-bb95-4ff676613c51-kube-api-access-v97dp\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:05:05.989812 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:05.989805    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f93340a-0884-496b-9a37-c9406e7ffccf-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:05:05.989812 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:05.989816    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f8vz\" (UniqueName: \"kubernetes.io/projected/4f93340a-0884-496b-9a37-c9406e7ffccf-kube-api-access-4f8vz\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:05:05.989812 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:05.989825    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cad37a3-e6bf-4b7e-bb95-4ff676613c51-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:05:06.384882 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:06.384845    2561 generic.go:356] "Generic (PLEG): container finished" podID="9cad37a3-e6bf-4b7e-bb95-4ff676613c51" containerID="60bd80ceb2ec74cd851e6fc3490a9215d3f7b3a7ad071a4ae23d31f18ee3e83d" exitCode=0
Jul 28 12:05:06.385070 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:06.384893    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9"
Jul 28 12:05:06.385070 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:06.384923    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9" event={"ID":"9cad37a3-e6bf-4b7e-bb95-4ff676613c51","Type":"ContainerDied","Data":"60bd80ceb2ec74cd851e6fc3490a9215d3f7b3a7ad071a4ae23d31f18ee3e83d"}
Jul 28 12:05:06.385070 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:06.384987    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9" event={"ID":"9cad37a3-e6bf-4b7e-bb95-4ff676613c51","Type":"ContainerDied","Data":"94e9ae76feadb7100e6923ff945bf889201edf0de7a91e9ef70fbe62807f2361"}
Jul 28 12:05:06.385070 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:06.385020    2561 scope.go:117] "RemoveContainer" containerID="60bd80ceb2ec74cd851e6fc3490a9215d3f7b3a7ad071a4ae23d31f18ee3e83d"
Jul 28 12:05:06.386263 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:06.386237    2561 generic.go:356] "Generic (PLEG): container finished" podID="4f93340a-0884-496b-9a37-c9406e7ffccf" containerID="082ea7712e8defd76195ba83ce513c2937756ce428d983c656c730af88a72a0f" exitCode=0
Jul 28 12:05:06.386380 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:06.386280    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn"
Jul 28 12:05:06.386380 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:06.386274    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn" event={"ID":"4f93340a-0884-496b-9a37-c9406e7ffccf","Type":"ContainerDied","Data":"082ea7712e8defd76195ba83ce513c2937756ce428d983c656c730af88a72a0f"}
Jul 28 12:05:06.386380 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:06.386372    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn" event={"ID":"4f93340a-0884-496b-9a37-c9406e7ffccf","Type":"ContainerDied","Data":"1c83d0c73263d7659ebfc4a82841a583eb02a15b2cda150202fdc0069e82e3a4"}
Jul 28 12:05:06.393528 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:06.393512    2561 scope.go:117] "RemoveContainer" containerID="60bd80ceb2ec74cd851e6fc3490a9215d3f7b3a7ad071a4ae23d31f18ee3e83d"
Jul 28 12:05:06.393795 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:05:06.393775    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60bd80ceb2ec74cd851e6fc3490a9215d3f7b3a7ad071a4ae23d31f18ee3e83d\": container with ID starting with 60bd80ceb2ec74cd851e6fc3490a9215d3f7b3a7ad071a4ae23d31f18ee3e83d not found: ID does not exist" containerID="60bd80ceb2ec74cd851e6fc3490a9215d3f7b3a7ad071a4ae23d31f18ee3e83d"
Jul 28 12:05:06.393893 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:06.393805    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60bd80ceb2ec74cd851e6fc3490a9215d3f7b3a7ad071a4ae23d31f18ee3e83d"} err="failed to get container status \"60bd80ceb2ec74cd851e6fc3490a9215d3f7b3a7ad071a4ae23d31f18ee3e83d\": rpc error: code = NotFound desc = could not find container \"60bd80ceb2ec74cd851e6fc3490a9215d3f7b3a7ad071a4ae23d31f18ee3e83d\": container with ID starting with 60bd80ceb2ec74cd851e6fc3490a9215d3f7b3a7ad071a4ae23d31f18ee3e83d not found: ID does not exist"
Jul 28 12:05:06.393893 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:06.393826    2561 scope.go:117] "RemoveContainer" containerID="082ea7712e8defd76195ba83ce513c2937756ce428d983c656c730af88a72a0f"
Jul 28 12:05:06.401188 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:06.401171    2561 scope.go:117] "RemoveContainer" containerID="082ea7712e8defd76195ba83ce513c2937756ce428d983c656c730af88a72a0f"
Jul 28 12:05:06.401455 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:05:06.401412    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"082ea7712e8defd76195ba83ce513c2937756ce428d983c656c730af88a72a0f\": container with ID starting with 082ea7712e8defd76195ba83ce513c2937756ce428d983c656c730af88a72a0f not found: ID does not exist" containerID="082ea7712e8defd76195ba83ce513c2937756ce428d983c656c730af88a72a0f"
Jul 28 12:05:06.401455 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:06.401439    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"082ea7712e8defd76195ba83ce513c2937756ce428d983c656c730af88a72a0f"} err="failed to get container status \"082ea7712e8defd76195ba83ce513c2937756ce428d983c656c730af88a72a0f\": rpc error: code = NotFound desc = could not find container \"082ea7712e8defd76195ba83ce513c2937756ce428d983c656c730af88a72a0f\": container with ID starting with 082ea7712e8defd76195ba83ce513c2937756ce428d983c656c730af88a72a0f not found: ID does not exist"
Jul 28 12:05:06.415156 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:06.415134    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9"]
Jul 28 12:05:06.419498 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:06.419480    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c98bc341-5d8f8f57f6-kpsv9"]
Jul 28 12:05:06.432886 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:06.432866    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn"]
Jul 28 12:05:06.445058 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:06.445039    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c98bc341-86c885c449-d2bgn"]
Jul 28 12:05:06.545812 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:06.545784    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f93340a-0884-496b-9a37-c9406e7ffccf" path="/var/lib/kubelet/pods/4f93340a-0884-496b-9a37-c9406e7ffccf/volumes"
Jul 28 12:05:06.546134 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:06.546122    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cad37a3-e6bf-4b7e-bb95-4ff676613c51" path="/var/lib/kubelet/pods/9cad37a3-e6bf-4b7e-bb95-4ff676613c51/volumes"
Jul 28 12:05:08.442322 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:08.442294    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:05:13.605944 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.605869    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"]
Jul 28 12:05:13.606342 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.606162    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f93340a-0884-496b-9a37-c9406e7ffccf" containerName="payload-processing"
Jul 28 12:05:13.606342 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.606173    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f93340a-0884-496b-9a37-c9406e7ffccf" containerName="payload-processing"
Jul 28 12:05:13.606342 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.606193    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cad37a3-e6bf-4b7e-bb95-4ff676613c51" containerName="payload-pre-processing"
Jul 28 12:05:13.606342 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.606198    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cad37a3-e6bf-4b7e-bb95-4ff676613c51" containerName="payload-pre-processing"
Jul 28 12:05:13.606342 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.606246    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cad37a3-e6bf-4b7e-bb95-4ff676613c51" containerName="payload-pre-processing"
Jul 28 12:05:13.606342 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.606253    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f93340a-0884-496b-9a37-c9406e7ffccf" containerName="payload-processing"
Jul 28 12:05:13.610415 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.610395    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"
Jul 28 12:05:13.613974 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.613953    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xchcw\""
Jul 28 12:05:13.614362 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.614335    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 12:05:13.614479 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.614457    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 12:05:13.614591 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.614565    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ae7b06d1-serving-cert\""
Jul 28 12:05:13.625244 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.625222    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"]
Jul 28 12:05:13.643888 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.643864    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66lgp\" (UniqueName: \"kubernetes.io/projected/584f2980-c301-48c0-98f9-49a87a12063f-kube-api-access-66lgp\") pod \"maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv\" (UID: \"584f2980-c301-48c0-98f9-49a87a12063f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"
Jul 28 12:05:13.643987 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.643901    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/584f2980-c301-48c0-98f9-49a87a12063f-openshift-service-ca\") pod \"maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv\" (UID: \"584f2980-c301-48c0-98f9-49a87a12063f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"
Jul 28 12:05:13.644036 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.644017    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/584f2980-c301-48c0-98f9-49a87a12063f-maas-api-tls\") pod \"maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv\" (UID: \"584f2980-c301-48c0-98f9-49a87a12063f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"
Jul 28 12:05:13.745203 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.745172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66lgp\" (UniqueName: \"kubernetes.io/projected/584f2980-c301-48c0-98f9-49a87a12063f-kube-api-access-66lgp\") pod \"maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv\" (UID: \"584f2980-c301-48c0-98f9-49a87a12063f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"
Jul 28 12:05:13.745203 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.745205    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/584f2980-c301-48c0-98f9-49a87a12063f-openshift-service-ca\") pod \"maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv\" (UID: \"584f2980-c301-48c0-98f9-49a87a12063f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"
Jul 28 12:05:13.745415 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.745258    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/584f2980-c301-48c0-98f9-49a87a12063f-maas-api-tls\") pod \"maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv\" (UID: \"584f2980-c301-48c0-98f9-49a87a12063f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"
Jul 28 12:05:13.745924 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.745900    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/584f2980-c301-48c0-98f9-49a87a12063f-openshift-service-ca\") pod \"maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv\" (UID: \"584f2980-c301-48c0-98f9-49a87a12063f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"
Jul 28 12:05:13.747540 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.747520    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/584f2980-c301-48c0-98f9-49a87a12063f-maas-api-tls\") pod \"maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv\" (UID: \"584f2980-c301-48c0-98f9-49a87a12063f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"
Jul 28 12:05:13.753111 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.753086    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66lgp\" (UniqueName: \"kubernetes.io/projected/584f2980-c301-48c0-98f9-49a87a12063f-kube-api-access-66lgp\") pod \"maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv\" (UID: \"584f2980-c301-48c0-98f9-49a87a12063f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"
Jul 28 12:05:13.925949 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:13.925872    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"
Jul 28 12:05:14.058676 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.058648    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"]
Jul 28 12:05:14.060637 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:05:14.060608    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584f2980_c301_48c0_98f9_49a87a12063f.slice/crio-56d1e38de194f1bbf0c568f73367b7a612fac239320b0e8d44e49363ea9c4069 WatchSource:0}: Error finding container 56d1e38de194f1bbf0c568f73367b7a612fac239320b0e8d44e49363ea9c4069: Status 404 returned error can't find the container with id 56d1e38de194f1bbf0c568f73367b7a612fac239320b0e8d44e49363ea9c4069
Jul 28 12:05:14.062047 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.062024    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:05:14.428498 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.428464    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv" event={"ID":"584f2980-c301-48c0-98f9-49a87a12063f","Type":"ContainerStarted","Data":"56d1e38de194f1bbf0c568f73367b7a612fac239320b0e8d44e49363ea9c4069"}
Jul 28 12:05:14.470315 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.470282    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq"]
Jul 28 12:05:14.474785 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.474766    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq"
Jul 28 12:05:14.476878 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.476858    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 12:05:14.476972 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.476858    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ae7b06d1\""
Jul 28 12:05:14.477173 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.477142    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ae7b06d1-dockercfg-6zx68\""
Jul 28 12:05:14.477512 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.477494    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 12:05:14.482302 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.482254    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq"]
Jul 28 12:05:14.553272 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.553249    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzqmm\" (UniqueName: \"kubernetes.io/projected/157f42f8-c91b-4a78-b5de-53225504347c-kube-api-access-vzqmm\") pod \"payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq\" (UID: \"157f42f8-c91b-4a78-b5de-53225504347c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq"
Jul 28 12:05:14.553400 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.553304    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/157f42f8-c91b-4a78-b5de-53225504347c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq\" (UID: \"157f42f8-c91b-4a78-b5de-53225504347c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq"
Jul 28 12:05:14.591513 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.591488    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b"]
Jul 28 12:05:14.594684 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.594669    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b"
Jul 28 12:05:14.607464 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.607444    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b"]
Jul 28 12:05:14.654179 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.654156    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6439a17b-f849-4ad5-97f3-aa8f52f83365-plugins-config-volume\") pod \"payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b\" (UID: \"6439a17b-f849-4ad5-97f3-aa8f52f83365\") " pod="openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b"
Jul 28 12:05:14.654300 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.654193    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzqmm\" (UniqueName: \"kubernetes.io/projected/157f42f8-c91b-4a78-b5de-53225504347c-kube-api-access-vzqmm\") pod \"payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq\" (UID: \"157f42f8-c91b-4a78-b5de-53225504347c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq"
Jul 28 12:05:14.654300 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.654232    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/157f42f8-c91b-4a78-b5de-53225504347c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq\" (UID: \"157f42f8-c91b-4a78-b5de-53225504347c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq"
Jul 28 12:05:14.654300 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.654249    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfp6n\" (UniqueName: \"kubernetes.io/projected/6439a17b-f849-4ad5-97f3-aa8f52f83365-kube-api-access-kfp6n\") pod \"payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b\" (UID: \"6439a17b-f849-4ad5-97f3-aa8f52f83365\") " pod="openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b"
Jul 28 12:05:14.654869 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.654852    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/157f42f8-c91b-4a78-b5de-53225504347c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq\" (UID: \"157f42f8-c91b-4a78-b5de-53225504347c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq"
Jul 28 12:05:14.662926 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.662908    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzqmm\" (UniqueName: \"kubernetes.io/projected/157f42f8-c91b-4a78-b5de-53225504347c-kube-api-access-vzqmm\") pod \"payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq\" (UID: \"157f42f8-c91b-4a78-b5de-53225504347c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq"
Jul 28 12:05:14.755330 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.755256    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6439a17b-f849-4ad5-97f3-aa8f52f83365-plugins-config-volume\") pod \"payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b\" (UID: \"6439a17b-f849-4ad5-97f3-aa8f52f83365\") " pod="openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b"
Jul 28 12:05:14.755444 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.755329    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfp6n\" (UniqueName: \"kubernetes.io/projected/6439a17b-f849-4ad5-97f3-aa8f52f83365-kube-api-access-kfp6n\") pod \"payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b\" (UID: \"6439a17b-f849-4ad5-97f3-aa8f52f83365\") " pod="openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b"
Jul 28 12:05:14.756043 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.756022    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6439a17b-f849-4ad5-97f3-aa8f52f83365-plugins-config-volume\") pod \"payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b\" (UID: \"6439a17b-f849-4ad5-97f3-aa8f52f83365\") " pod="openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b"
Jul 28 12:05:14.766611 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.766565    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfp6n\" (UniqueName: \"kubernetes.io/projected/6439a17b-f849-4ad5-97f3-aa8f52f83365-kube-api-access-kfp6n\") pod \"payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b\" (UID: \"6439a17b-f849-4ad5-97f3-aa8f52f83365\") " pod="openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b"
Jul 28 12:05:14.785888 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.785866    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq"
Jul 28 12:05:14.904113 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.904082    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b"
Jul 28 12:05:14.917462 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:14.917441    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq"]
Jul 28 12:05:14.920649 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:05:14.920614    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157f42f8_c91b_4a78_b5de_53225504347c.slice/crio-ca8adf274ab5c6321602b72e5b975bafa8f7ce956256d6ea999d8b510faf1550 WatchSource:0}: Error finding container ca8adf274ab5c6321602b72e5b975bafa8f7ce956256d6ea999d8b510faf1550: Status 404 returned error can't find the container with id ca8adf274ab5c6321602b72e5b975bafa8f7ce956256d6ea999d8b510faf1550
Jul 28 12:05:15.037627 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:15.037589    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b"]
Jul 28 12:05:15.040020 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:05:15.039984    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6439a17b_f849_4ad5_97f3_aa8f52f83365.slice/crio-aa975c684067d16636687d34e1007a95c64226dab86abe7af2a01e63dae867c6 WatchSource:0}: Error finding container aa975c684067d16636687d34e1007a95c64226dab86abe7af2a01e63dae867c6: Status 404 returned error can't find the container with id aa975c684067d16636687d34e1007a95c64226dab86abe7af2a01e63dae867c6
Jul 28 12:05:15.435077 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:15.435038    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b" event={"ID":"6439a17b-f849-4ad5-97f3-aa8f52f83365","Type":"ContainerStarted","Data":"cfbcdc23a35c39fa71ed27664fedec0756091e2e541d4b99b6fa637b236a629e"}
Jul 28 12:05:15.435249 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:15.435090    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b" event={"ID":"6439a17b-f849-4ad5-97f3-aa8f52f83365","Type":"ContainerStarted","Data":"aa975c684067d16636687d34e1007a95c64226dab86abe7af2a01e63dae867c6"}
Jul 28 12:05:15.435249 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:15.435158    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b"
Jul 28 12:05:15.436638 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:15.436610    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq" event={"ID":"157f42f8-c91b-4a78-b5de-53225504347c","Type":"ContainerStarted","Data":"658d5d3da0942159b2f14a805f7aacca63663a36e0db8b9d8f4b71b7daa0fed3"}
Jul 28 12:05:15.436763 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:15.436656    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq" event={"ID":"157f42f8-c91b-4a78-b5de-53225504347c","Type":"ContainerStarted","Data":"ca8adf274ab5c6321602b72e5b975bafa8f7ce956256d6ea999d8b510faf1550"}
Jul 28 12:05:15.436763 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:15.436686    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq"
Jul 28 12:05:15.451293 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:15.451215    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b" podStartSLOduration=1.4512014770000001 podStartE2EDuration="1.451201477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:05:15.449529223 +0000 UTC m=+2067.308185222" watchObservedRunningTime="2026-07-28 12:05:15.451201477 +0000 UTC m=+2067.309857475"
Jul 28 12:05:15.465844 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:15.465443    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq" podStartSLOduration=1.465427922 podStartE2EDuration="1.465427922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:05:15.464628817 +0000 UTC m=+2067.323284817" watchObservedRunningTime="2026-07-28 12:05:15.465427922 +0000 UTC m=+2067.324083931"
Jul 28 12:05:17.448786 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:17.448758    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv" event={"ID":"584f2980-c301-48c0-98f9-49a87a12063f","Type":"ContainerStarted","Data":"1c12d4684f24ed0bdec5612a81cee3afa42c05b605c90bc2106d2116153ddfc9"}
Jul 28 12:05:17.449115 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:17.448871    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"
Jul 28 12:05:17.469925 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:17.469879    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv" podStartSLOduration=1.219431707 podStartE2EDuration="4.469858467s" totalImagesPullingTime="3.25042676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:05:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:05:17.468285405 +0000 UTC m=+2069.326941405" watchObservedRunningTime="2026-07-28 12:05:17.469858467 +0000 UTC m=+2069.328514467"
Jul 28 12:05:19.739558 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:19.739524    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:05:23.457879 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:23.457851    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"
Jul 28 12:05:26.445325 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:26.445294    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b"
Jul 28 12:05:26.445808 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:26.445366    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq"
Jul 28 12:05:49.873285 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:49.873242    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"]
Jul 28 12:05:49.873795 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:49.873492    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv" podUID="584f2980-c301-48c0-98f9-49a87a12063f" containerName="maas-api" containerID="cri-o://1c12d4684f24ed0bdec5612a81cee3afa42c05b605c90bc2106d2116153ddfc9" gracePeriod=30
Jul 28 12:05:49.930405 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:49.929856    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b"]
Jul 28 12:05:49.930405 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:49.930082    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b" podUID="6439a17b-f849-4ad5-97f3-aa8f52f83365" containerName="payload-processing" containerID="cri-o://cfbcdc23a35c39fa71ed27664fedec0756091e2e541d4b99b6fa637b236a629e" gracePeriod=30
Jul 28 12:05:49.935759 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:49.935724    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq"]
Jul 28 12:05:49.936003 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:49.935930    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq" podUID="157f42f8-c91b-4a78-b5de-53225504347c" containerName="payload-pre-processing" containerID="cri-o://658d5d3da0942159b2f14a805f7aacca63663a36e0db8b9d8f4b71b7daa0fed3" gracePeriod=30
Jul 28 12:05:50.202371 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.202350    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"
Jul 28 12:05:50.205329 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.205307    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b"
Jul 28 12:05:50.221856 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.221791    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq"
Jul 28 12:05:50.347313 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.347283    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/584f2980-c301-48c0-98f9-49a87a12063f-openshift-service-ca\") pod \"584f2980-c301-48c0-98f9-49a87a12063f\" (UID: \"584f2980-c301-48c0-98f9-49a87a12063f\") "
Jul 28 12:05:50.347467 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.347325    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6439a17b-f849-4ad5-97f3-aa8f52f83365-plugins-config-volume\") pod \"6439a17b-f849-4ad5-97f3-aa8f52f83365\" (UID: \"6439a17b-f849-4ad5-97f3-aa8f52f83365\") "
Jul 28 12:05:50.347467 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.347364    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/157f42f8-c91b-4a78-b5de-53225504347c-plugins-config-volume\") pod \"157f42f8-c91b-4a78-b5de-53225504347c\" (UID: \"157f42f8-c91b-4a78-b5de-53225504347c\") "
Jul 28 12:05:50.347467 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.347405    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzqmm\" (UniqueName: \"kubernetes.io/projected/157f42f8-c91b-4a78-b5de-53225504347c-kube-api-access-vzqmm\") pod \"157f42f8-c91b-4a78-b5de-53225504347c\" (UID: \"157f42f8-c91b-4a78-b5de-53225504347c\") "
Jul 28 12:05:50.347467 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.347461    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66lgp\" (UniqueName: \"kubernetes.io/projected/584f2980-c301-48c0-98f9-49a87a12063f-kube-api-access-66lgp\") pod \"584f2980-c301-48c0-98f9-49a87a12063f\" (UID: \"584f2980-c301-48c0-98f9-49a87a12063f\") "
Jul 28 12:05:50.347709 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.347496    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfp6n\" (UniqueName: \"kubernetes.io/projected/6439a17b-f849-4ad5-97f3-aa8f52f83365-kube-api-access-kfp6n\") pod \"6439a17b-f849-4ad5-97f3-aa8f52f83365\" (UID: \"6439a17b-f849-4ad5-97f3-aa8f52f83365\") "
Jul 28 12:05:50.347709 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.347524    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/584f2980-c301-48c0-98f9-49a87a12063f-maas-api-tls\") pod \"584f2980-c301-48c0-98f9-49a87a12063f\" (UID: \"584f2980-c301-48c0-98f9-49a87a12063f\") "
Jul 28 12:05:50.347813 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.347719    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/584f2980-c301-48c0-98f9-49a87a12063f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "584f2980-c301-48c0-98f9-49a87a12063f" (UID: "584f2980-c301-48c0-98f9-49a87a12063f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:05:50.347813 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.347791    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6439a17b-f849-4ad5-97f3-aa8f52f83365-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6439a17b-f849-4ad5-97f3-aa8f52f83365" (UID: "6439a17b-f849-4ad5-97f3-aa8f52f83365"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:05:50.347903 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.347845    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/584f2980-c301-48c0-98f9-49a87a12063f-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:05:50.348140 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.348114    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157f42f8-c91b-4a78-b5de-53225504347c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "157f42f8-c91b-4a78-b5de-53225504347c" (UID: "157f42f8-c91b-4a78-b5de-53225504347c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:05:50.349781 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.349741    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6439a17b-f849-4ad5-97f3-aa8f52f83365-kube-api-access-kfp6n" (OuterVolumeSpecName: "kube-api-access-kfp6n") pod "6439a17b-f849-4ad5-97f3-aa8f52f83365" (UID: "6439a17b-f849-4ad5-97f3-aa8f52f83365"). InnerVolumeSpecName "kube-api-access-kfp6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:05:50.349879 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.349785    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157f42f8-c91b-4a78-b5de-53225504347c-kube-api-access-vzqmm" (OuterVolumeSpecName: "kube-api-access-vzqmm") pod "157f42f8-c91b-4a78-b5de-53225504347c" (UID: "157f42f8-c91b-4a78-b5de-53225504347c"). InnerVolumeSpecName "kube-api-access-vzqmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:05:50.349879 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.349838    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/584f2980-c301-48c0-98f9-49a87a12063f-kube-api-access-66lgp" (OuterVolumeSpecName: "kube-api-access-66lgp") pod "584f2980-c301-48c0-98f9-49a87a12063f" (UID: "584f2980-c301-48c0-98f9-49a87a12063f"). InnerVolumeSpecName "kube-api-access-66lgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:05:50.349879 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.349848    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/584f2980-c301-48c0-98f9-49a87a12063f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "584f2980-c301-48c0-98f9-49a87a12063f" (UID: "584f2980-c301-48c0-98f9-49a87a12063f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:05:50.449198 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.449129    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66lgp\" (UniqueName: \"kubernetes.io/projected/584f2980-c301-48c0-98f9-49a87a12063f-kube-api-access-66lgp\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:05:50.449198 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.449159    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfp6n\" (UniqueName: \"kubernetes.io/projected/6439a17b-f849-4ad5-97f3-aa8f52f83365-kube-api-access-kfp6n\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:05:50.449198 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.449173    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/584f2980-c301-48c0-98f9-49a87a12063f-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:05:50.449198 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.449185    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6439a17b-f849-4ad5-97f3-aa8f52f83365-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:05:50.449198 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.449198    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/157f42f8-c91b-4a78-b5de-53225504347c-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:05:50.449427 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.449211    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzqmm\" (UniqueName: \"kubernetes.io/projected/157f42f8-c91b-4a78-b5de-53225504347c-kube-api-access-vzqmm\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:05:50.593069 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.593044    2561 generic.go:356] "Generic (PLEG): container finished" podID="6439a17b-f849-4ad5-97f3-aa8f52f83365" containerID="cfbcdc23a35c39fa71ed27664fedec0756091e2e541d4b99b6fa637b236a629e" exitCode=0
Jul 28 12:05:50.593168 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.593086    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b"
Jul 28 12:05:50.593168 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.593109    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b" event={"ID":"6439a17b-f849-4ad5-97f3-aa8f52f83365","Type":"ContainerDied","Data":"cfbcdc23a35c39fa71ed27664fedec0756091e2e541d4b99b6fa637b236a629e"}
Jul 28 12:05:50.593168 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.593160    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b" event={"ID":"6439a17b-f849-4ad5-97f3-aa8f52f83365","Type":"ContainerDied","Data":"aa975c684067d16636687d34e1007a95c64226dab86abe7af2a01e63dae867c6"}
Jul 28 12:05:50.593310 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.593186    2561 scope.go:117] "RemoveContainer" containerID="cfbcdc23a35c39fa71ed27664fedec0756091e2e541d4b99b6fa637b236a629e"
Jul 28 12:05:50.594488 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.594463    2561 generic.go:356] "Generic (PLEG): container finished" podID="157f42f8-c91b-4a78-b5de-53225504347c" containerID="658d5d3da0942159b2f14a805f7aacca63663a36e0db8b9d8f4b71b7daa0fed3" exitCode=0
Jul 28 12:05:50.594596 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.594523    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq"
Jul 28 12:05:50.594596 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.594561    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq" event={"ID":"157f42f8-c91b-4a78-b5de-53225504347c","Type":"ContainerDied","Data":"658d5d3da0942159b2f14a805f7aacca63663a36e0db8b9d8f4b71b7daa0fed3"}
Jul 28 12:05:50.594717 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.594617    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq" event={"ID":"157f42f8-c91b-4a78-b5de-53225504347c","Type":"ContainerDied","Data":"ca8adf274ab5c6321602b72e5b975bafa8f7ce956256d6ea999d8b510faf1550"}
Jul 28 12:05:50.596017 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.595991    2561 generic.go:356] "Generic (PLEG): container finished" podID="584f2980-c301-48c0-98f9-49a87a12063f" containerID="1c12d4684f24ed0bdec5612a81cee3afa42c05b605c90bc2106d2116153ddfc9" exitCode=0
Jul 28 12:05:50.596113 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.596025    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv" event={"ID":"584f2980-c301-48c0-98f9-49a87a12063f","Type":"ContainerDied","Data":"1c12d4684f24ed0bdec5612a81cee3afa42c05b605c90bc2106d2116153ddfc9"}
Jul 28 12:05:50.596113 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.596040    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"
Jul 28 12:05:50.596113 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.596076    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv" event={"ID":"584f2980-c301-48c0-98f9-49a87a12063f","Type":"ContainerDied","Data":"56d1e38de194f1bbf0c568f73367b7a612fac239320b0e8d44e49363ea9c4069"}
Jul 28 12:05:50.602076 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.602058    2561 scope.go:117] "RemoveContainer" containerID="cfbcdc23a35c39fa71ed27664fedec0756091e2e541d4b99b6fa637b236a629e"
Jul 28 12:05:50.607500 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:05:50.602316    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfbcdc23a35c39fa71ed27664fedec0756091e2e541d4b99b6fa637b236a629e\": container with ID starting with cfbcdc23a35c39fa71ed27664fedec0756091e2e541d4b99b6fa637b236a629e not found: ID does not exist" containerID="cfbcdc23a35c39fa71ed27664fedec0756091e2e541d4b99b6fa637b236a629e"
Jul 28 12:05:50.607500 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.602684    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfbcdc23a35c39fa71ed27664fedec0756091e2e541d4b99b6fa637b236a629e"} err="failed to get container status \"cfbcdc23a35c39fa71ed27664fedec0756091e2e541d4b99b6fa637b236a629e\": rpc error: code = NotFound desc = could not find container \"cfbcdc23a35c39fa71ed27664fedec0756091e2e541d4b99b6fa637b236a629e\": container with ID starting with cfbcdc23a35c39fa71ed27664fedec0756091e2e541d4b99b6fa637b236a629e not found: ID does not exist"
Jul 28 12:05:50.607500 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.602712    2561 scope.go:117] "RemoveContainer" containerID="658d5d3da0942159b2f14a805f7aacca63663a36e0db8b9d8f4b71b7daa0fed3"
Jul 28 12:05:50.614770 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.614740    2561 scope.go:117] "RemoveContainer" containerID="658d5d3da0942159b2f14a805f7aacca63663a36e0db8b9d8f4b71b7daa0fed3"
Jul 28 12:05:50.615107 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:05:50.615077    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"658d5d3da0942159b2f14a805f7aacca63663a36e0db8b9d8f4b71b7daa0fed3\": container with ID starting with 658d5d3da0942159b2f14a805f7aacca63663a36e0db8b9d8f4b71b7daa0fed3 not found: ID does not exist" containerID="658d5d3da0942159b2f14a805f7aacca63663a36e0db8b9d8f4b71b7daa0fed3"
Jul 28 12:05:50.615176 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.615128    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658d5d3da0942159b2f14a805f7aacca63663a36e0db8b9d8f4b71b7daa0fed3"} err="failed to get container status \"658d5d3da0942159b2f14a805f7aacca63663a36e0db8b9d8f4b71b7daa0fed3\": rpc error: code = NotFound desc = could not find container \"658d5d3da0942159b2f14a805f7aacca63663a36e0db8b9d8f4b71b7daa0fed3\": container with ID starting with 658d5d3da0942159b2f14a805f7aacca63663a36e0db8b9d8f4b71b7daa0fed3 not found: ID does not exist"
Jul 28 12:05:50.615176 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.615164    2561 scope.go:117] "RemoveContainer" containerID="1c12d4684f24ed0bdec5612a81cee3afa42c05b605c90bc2106d2116153ddfc9"
Jul 28 12:05:50.617714 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.617698    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"]
Jul 28 12:05:50.621014 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.620998    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae7b06d1-5c4cf748f4-hb4kv"]
Jul 28 12:05:50.623571 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.623557    2561 scope.go:117] "RemoveContainer" containerID="1c12d4684f24ed0bdec5612a81cee3afa42c05b605c90bc2106d2116153ddfc9"
Jul 28 12:05:50.623815 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:05:50.623797    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c12d4684f24ed0bdec5612a81cee3afa42c05b605c90bc2106d2116153ddfc9\": container with ID starting with 1c12d4684f24ed0bdec5612a81cee3afa42c05b605c90bc2106d2116153ddfc9 not found: ID does not exist" containerID="1c12d4684f24ed0bdec5612a81cee3afa42c05b605c90bc2106d2116153ddfc9"
Jul 28 12:05:50.623881 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.623824    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c12d4684f24ed0bdec5612a81cee3afa42c05b605c90bc2106d2116153ddfc9"} err="failed to get container status \"1c12d4684f24ed0bdec5612a81cee3afa42c05b605c90bc2106d2116153ddfc9\": rpc error: code = NotFound desc = could not find container \"1c12d4684f24ed0bdec5612a81cee3afa42c05b605c90bc2106d2116153ddfc9\": container with ID starting with 1c12d4684f24ed0bdec5612a81cee3afa42c05b605c90bc2106d2116153ddfc9 not found: ID does not exist"
Jul 28 12:05:50.630509 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.630490    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq"]
Jul 28 12:05:50.634250 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.634230    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae7b06d1-58b5b77b8d-zbgzq"]
Jul 28 12:05:50.645013 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.644994    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b"]
Jul 28 12:05:50.648610 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:50.648570    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae7b06d1-6b58856864-8l74b"]
Jul 28 12:05:52.546025 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:52.545992    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="157f42f8-c91b-4a78-b5de-53225504347c" path="/var/lib/kubelet/pods/157f42f8-c91b-4a78-b5de-53225504347c/volumes"
Jul 28 12:05:52.546387 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:52.546367    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="584f2980-c301-48c0-98f9-49a87a12063f" path="/var/lib/kubelet/pods/584f2980-c301-48c0-98f9-49a87a12063f/volumes"
Jul 28 12:05:52.546690 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:52.546677    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6439a17b-f849-4ad5-97f3-aa8f52f83365" path="/var/lib/kubelet/pods/6439a17b-f849-4ad5-97f3-aa8f52f83365/volumes"
Jul 28 12:05:52.823218 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:52.823190    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:05:58.892592 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:58.892539    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"]
Jul 28 12:05:58.893044 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:58.892993    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="157f42f8-c91b-4a78-b5de-53225504347c" containerName="payload-pre-processing"
Jul 28 12:05:58.893044 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:58.893013    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="157f42f8-c91b-4a78-b5de-53225504347c" containerName="payload-pre-processing"
Jul 28 12:05:58.893044 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:58.893024    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="584f2980-c301-48c0-98f9-49a87a12063f" containerName="maas-api"
Jul 28 12:05:58.893044 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:58.893032    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="584f2980-c301-48c0-98f9-49a87a12063f" containerName="maas-api"
Jul 28 12:05:58.893257 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:58.893053    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6439a17b-f849-4ad5-97f3-aa8f52f83365" containerName="payload-processing"
Jul 28 12:05:58.893257 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:58.893062    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6439a17b-f849-4ad5-97f3-aa8f52f83365" containerName="payload-processing"
Jul 28 12:05:58.893257 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:58.893143    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="584f2980-c301-48c0-98f9-49a87a12063f" containerName="maas-api"
Jul 28 12:05:58.893257 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:58.893165    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="157f42f8-c91b-4a78-b5de-53225504347c" containerName="payload-pre-processing"
Jul 28 12:05:58.893257 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:58.893175    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6439a17b-f849-4ad5-97f3-aa8f52f83365" containerName="payload-processing"
Jul 28 12:05:58.896384 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:58.896363    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"
Jul 28 12:05:58.898407 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:58.898383    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xchcw\""
Jul 28 12:05:58.898523 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:58.898458    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 12:05:58.899028 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:58.899009    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 12:05:58.899082 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:58.899019    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-9636f5f9-serving-cert\""
Jul 28 12:05:58.907860 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:58.907837    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"]
Jul 28 12:05:59.016709 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.016679    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9rgh\" (UniqueName: \"kubernetes.io/projected/0306a9f3-77bd-4bfd-a635-e62cca900029-kube-api-access-n9rgh\") pod \"maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq\" (UID: \"0306a9f3-77bd-4bfd-a635-e62cca900029\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"
Jul 28 12:05:59.016709 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.016713    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0306a9f3-77bd-4bfd-a635-e62cca900029-maas-api-tls\") pod \"maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq\" (UID: \"0306a9f3-77bd-4bfd-a635-e62cca900029\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"
Jul 28 12:05:59.016912 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.016784    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0306a9f3-77bd-4bfd-a635-e62cca900029-openshift-service-ca\") pod \"maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq\" (UID: \"0306a9f3-77bd-4bfd-a635-e62cca900029\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"
Jul 28 12:05:59.117854 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.117823    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9rgh\" (UniqueName: \"kubernetes.io/projected/0306a9f3-77bd-4bfd-a635-e62cca900029-kube-api-access-n9rgh\") pod \"maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq\" (UID: \"0306a9f3-77bd-4bfd-a635-e62cca900029\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"
Jul 28 12:05:59.117854 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.117856    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0306a9f3-77bd-4bfd-a635-e62cca900029-maas-api-tls\") pod \"maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq\" (UID: \"0306a9f3-77bd-4bfd-a635-e62cca900029\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"
Jul 28 12:05:59.118040 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.117892    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0306a9f3-77bd-4bfd-a635-e62cca900029-openshift-service-ca\") pod \"maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq\" (UID: \"0306a9f3-77bd-4bfd-a635-e62cca900029\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"
Jul 28 12:05:59.118040 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:05:59.117976    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-serving-cert: secret "maas-api-e2e-derive-9636f5f9-serving-cert" not found
Jul 28 12:05:59.118108 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:05:59.118051    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0306a9f3-77bd-4bfd-a635-e62cca900029-maas-api-tls podName:0306a9f3-77bd-4bfd-a635-e62cca900029 nodeName:}" failed. No retries permitted until 2026-07-28 12:05:59.618032824 +0000 UTC m=+2111.476688809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0306a9f3-77bd-4bfd-a635-e62cca900029-maas-api-tls") pod "maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq" (UID: "0306a9f3-77bd-4bfd-a635-e62cca900029") : secret "maas-api-e2e-derive-9636f5f9-serving-cert" not found
Jul 28 12:05:59.118476 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.118459    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0306a9f3-77bd-4bfd-a635-e62cca900029-openshift-service-ca\") pod \"maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq\" (UID: \"0306a9f3-77bd-4bfd-a635-e62cca900029\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"
Jul 28 12:05:59.137190 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.137169    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9rgh\" (UniqueName: \"kubernetes.io/projected/0306a9f3-77bd-4bfd-a635-e62cca900029-kube-api-access-n9rgh\") pod \"maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq\" (UID: \"0306a9f3-77bd-4bfd-a635-e62cca900029\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"
Jul 28 12:05:59.621731 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.621699    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0306a9f3-77bd-4bfd-a635-e62cca900029-maas-api-tls\") pod \"maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq\" (UID: \"0306a9f3-77bd-4bfd-a635-e62cca900029\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"
Jul 28 12:05:59.624123 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.624092    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0306a9f3-77bd-4bfd-a635-e62cca900029-maas-api-tls\") pod \"maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq\" (UID: \"0306a9f3-77bd-4bfd-a635-e62cca900029\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"
Jul 28 12:05:59.717360 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.717330    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc"]
Jul 28 12:05:59.720845 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.720830    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc"
Jul 28 12:05:59.723558 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.723529    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 12:05:59.723692 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.723563    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 12:05:59.724042 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.724026    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-9636f5f9-dockercfg-k8cw2\""
Jul 28 12:05:59.724134 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.724056    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-9636f5f9\""
Jul 28 12:05:59.727477 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.727455    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5"]
Jul 28 12:05:59.730831 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.730813    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5"
Jul 28 12:05:59.738662 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.738643    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc"]
Jul 28 12:05:59.750118 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.750091    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5"]
Jul 28 12:05:59.807940 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.807909    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"
Jul 28 12:05:59.823328 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.823300    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee6d965-ae63-4b8f-add7-b20def1eb982-plugins-config-volume\") pod \"payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5\" (UID: \"7ee6d965-ae63-4b8f-add7-b20def1eb982\") " pod="openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5"
Jul 28 12:05:59.823460 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.823375    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a687b09b-802d-4b4f-965c-df3d096d01be-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc\" (UID: \"a687b09b-802d-4b4f-965c-df3d096d01be\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc"
Jul 28 12:05:59.823460 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.823404    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8wv6\" (UniqueName: \"kubernetes.io/projected/7ee6d965-ae63-4b8f-add7-b20def1eb982-kube-api-access-c8wv6\") pod \"payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5\" (UID: \"7ee6d965-ae63-4b8f-add7-b20def1eb982\") " pod="openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5"
Jul 28 12:05:59.823648 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.823481    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6kqc\" (UniqueName: \"kubernetes.io/projected/a687b09b-802d-4b4f-965c-df3d096d01be-kube-api-access-x6kqc\") pod \"payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc\" (UID: \"a687b09b-802d-4b4f-965c-df3d096d01be\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc"
Jul 28 12:05:59.924371 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.924338    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6kqc\" (UniqueName: \"kubernetes.io/projected/a687b09b-802d-4b4f-965c-df3d096d01be-kube-api-access-x6kqc\") pod \"payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc\" (UID: \"a687b09b-802d-4b4f-965c-df3d096d01be\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc"
Jul 28 12:05:59.924743 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.924389    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee6d965-ae63-4b8f-add7-b20def1eb982-plugins-config-volume\") pod \"payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5\" (UID: \"7ee6d965-ae63-4b8f-add7-b20def1eb982\") " pod="openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5"
Jul 28 12:05:59.924743 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.924452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a687b09b-802d-4b4f-965c-df3d096d01be-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc\" (UID: \"a687b09b-802d-4b4f-965c-df3d096d01be\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc"
Jul 28 12:05:59.924743 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.924481    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8wv6\" (UniqueName: \"kubernetes.io/projected/7ee6d965-ae63-4b8f-add7-b20def1eb982-kube-api-access-c8wv6\") pod \"payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5\" (UID: \"7ee6d965-ae63-4b8f-add7-b20def1eb982\") " pod="openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5"
Jul 28 12:05:59.925116 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.925098    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a687b09b-802d-4b4f-965c-df3d096d01be-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc\" (UID: \"a687b09b-802d-4b4f-965c-df3d096d01be\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc"
Jul 28 12:05:59.925418 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.925391    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee6d965-ae63-4b8f-add7-b20def1eb982-plugins-config-volume\") pod \"payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5\" (UID: \"7ee6d965-ae63-4b8f-add7-b20def1eb982\") " pod="openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5"
Jul 28 12:05:59.933663 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.933626    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8wv6\" (UniqueName: \"kubernetes.io/projected/7ee6d965-ae63-4b8f-add7-b20def1eb982-kube-api-access-c8wv6\") pod \"payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5\" (UID: \"7ee6d965-ae63-4b8f-add7-b20def1eb982\") " pod="openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5"
Jul 28 12:05:59.934334 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.934310    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6kqc\" (UniqueName: \"kubernetes.io/projected/a687b09b-802d-4b4f-965c-df3d096d01be-kube-api-access-x6kqc\") pod \"payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc\" (UID: \"a687b09b-802d-4b4f-965c-df3d096d01be\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc"
Jul 28 12:05:59.941780 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:05:59.941759    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"]
Jul 28 12:05:59.945068 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:05:59.945046    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0306a9f3_77bd_4bfd_a635_e62cca900029.slice/crio-994a877e3a17352427cdff1851f143256c79ac60ae274e73b61afacd7420edde WatchSource:0}: Error finding container 994a877e3a17352427cdff1851f143256c79ac60ae274e73b61afacd7420edde: Status 404 returned error can't find the container with id 994a877e3a17352427cdff1851f143256c79ac60ae274e73b61afacd7420edde
Jul 28 12:06:00.031916 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:00.031889    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc"
Jul 28 12:06:00.041423 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:00.041402    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5"
Jul 28 12:06:00.170156 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:00.170116    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc"]
Jul 28 12:06:00.173541 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:06:00.173511    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda687b09b_802d_4b4f_965c_df3d096d01be.slice/crio-515c6ce0e17f778b0046d32d661d86628b2f1777fd3057d15a55e632395e1c74 WatchSource:0}: Error finding container 515c6ce0e17f778b0046d32d661d86628b2f1777fd3057d15a55e632395e1c74: Status 404 returned error can't find the container with id 515c6ce0e17f778b0046d32d661d86628b2f1777fd3057d15a55e632395e1c74
Jul 28 12:06:00.198692 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:00.198669    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5"]
Jul 28 12:06:00.216934 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:06:00.216904    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee6d965_ae63_4b8f_add7_b20def1eb982.slice/crio-5446dddc0b130c14c5fecd9424708f1f8070998fb72c1f1afe58ba39a6ff2d3a WatchSource:0}: Error finding container 5446dddc0b130c14c5fecd9424708f1f8070998fb72c1f1afe58ba39a6ff2d3a: Status 404 returned error can't find the container with id 5446dddc0b130c14c5fecd9424708f1f8070998fb72c1f1afe58ba39a6ff2d3a
Jul 28 12:06:00.640792 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:00.640751    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5" event={"ID":"7ee6d965-ae63-4b8f-add7-b20def1eb982","Type":"ContainerStarted","Data":"3a6aff812a24bffbe9979405064eda2a23994030109d7bf57c2685db987fc7ec"}
Jul 28 12:06:00.640953 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:00.640806    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5" event={"ID":"7ee6d965-ae63-4b8f-add7-b20def1eb982","Type":"ContainerStarted","Data":"5446dddc0b130c14c5fecd9424708f1f8070998fb72c1f1afe58ba39a6ff2d3a"}
Jul 28 12:06:00.640953 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:00.640867    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5"
Jul 28 12:06:00.642201 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:00.642180    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc" event={"ID":"a687b09b-802d-4b4f-965c-df3d096d01be","Type":"ContainerStarted","Data":"8d3494640294f5ee086f0f6d9a9971a2356f999021814489d7ef4985dd98c24e"}
Jul 28 12:06:00.642312 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:00.642215    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc" event={"ID":"a687b09b-802d-4b4f-965c-df3d096d01be","Type":"ContainerStarted","Data":"515c6ce0e17f778b0046d32d661d86628b2f1777fd3057d15a55e632395e1c74"}
Jul 28 12:06:00.642312 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:00.642263    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc"
Jul 28 12:06:00.643619 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:00.643598    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq" event={"ID":"0306a9f3-77bd-4bfd-a635-e62cca900029","Type":"ContainerStarted","Data":"641fabe1999467110a065c69d220ea0f86987d6ea3a00b65e809ec7cfc076339"}
Jul 28 12:06:00.643699 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:00.643633    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq" event={"ID":"0306a9f3-77bd-4bfd-a635-e62cca900029","Type":"ContainerStarted","Data":"994a877e3a17352427cdff1851f143256c79ac60ae274e73b61afacd7420edde"}
Jul 28 12:06:00.643699 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:00.643653    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"
Jul 28 12:06:00.657987 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:00.657945    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5" podStartSLOduration=1.657933136 podStartE2EDuration="1.657933136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:06:00.656251447 +0000 UTC m=+2112.514907447" watchObservedRunningTime="2026-07-28 12:06:00.657933136 +0000 UTC m=+2112.516589113"
Jul 28 12:06:00.672206 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:00.672164    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc" podStartSLOduration=1.67215098 podStartE2EDuration="1.67215098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:06:00.670951692 +0000 UTC m=+2112.529607692" watchObservedRunningTime="2026-07-28 12:06:00.67215098 +0000 UTC m=+2112.530806957"
Jul 28 12:06:00.690533 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:00.690493    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq" podStartSLOduration=2.153249627 podStartE2EDuration="2.690479899s" totalImagesPullingTime="537.230272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:06:00.688177041 +0000 UTC m=+2112.546833061" watchObservedRunningTime="2026-07-28 12:06:00.690479899 +0000 UTC m=+2112.549135898"
Jul 28 12:06:01.726871 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:01.726833    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:06:06.653847 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:06.653818    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"
Jul 28 12:06:11.650093 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:11.650057    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc"
Jul 28 12:06:11.650573 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:11.650450    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5"
Jul 28 12:06:23.573404 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:23.573371    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9636f5f9-tt7j9"]
Jul 28 12:06:23.576594 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:23.576564    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9636f5f9-tt7j9"
Jul 28 12:06:23.578961 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:23.578940    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ts4x9\""
Jul 28 12:06:23.583839 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:23.583817    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9636f5f9-tt7j9"]
Jul 28 12:06:23.614435 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:23.614410    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ee667f-0cd3-48fc-9336-e2f97e893e40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9636f5f9-tt7j9\" (UID: \"93ee667f-0cd3-48fc-9336-e2f97e893e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9636f5f9-tt7j9"
Jul 28 12:06:23.715068 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:23.715039    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ee667f-0cd3-48fc-9336-e2f97e893e40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9636f5f9-tt7j9\" (UID: \"93ee667f-0cd3-48fc-9336-e2f97e893e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9636f5f9-tt7j9"
Jul 28 12:06:23.715685 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:23.715664    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ee667f-0cd3-48fc-9336-e2f97e893e40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9636f5f9-tt7j9\" (UID: \"93ee667f-0cd3-48fc-9336-e2f97e893e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9636f5f9-tt7j9"
Jul 28 12:06:23.888847 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:23.888771    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9636f5f9-tt7j9"
Jul 28 12:06:24.017517 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:24.017489    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9636f5f9-tt7j9"]
Jul 28 12:06:24.019971 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:06:24.019942    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ee667f_0cd3_48fc_9336_e2f97e893e40.slice/crio-37cb1ae19f75ffec343c40410f92f46ab53f1121e89582d8effbe1d569623059 WatchSource:0}: Error finding container 37cb1ae19f75ffec343c40410f92f46ab53f1121e89582d8effbe1d569623059: Status 404 returned error can't find the container with id 37cb1ae19f75ffec343c40410f92f46ab53f1121e89582d8effbe1d569623059
Jul 28 12:06:24.751389 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:24.751355    2561 generic.go:356] "Generic (PLEG): container finished" podID="93ee667f-0cd3-48fc-9336-e2f97e893e40" containerID="c98cc3e687aa5b2d81c6919af0a16a4f24bf305821241778e681cd1af8e1b42b" exitCode=0
Jul 28 12:06:24.751801 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:24.751443    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9636f5f9-tt7j9" event={"ID":"93ee667f-0cd3-48fc-9336-e2f97e893e40","Type":"ContainerDied","Data":"c98cc3e687aa5b2d81c6919af0a16a4f24bf305821241778e681cd1af8e1b42b"}
Jul 28 12:06:24.751801 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:24.751497    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9636f5f9-tt7j9" event={"ID":"93ee667f-0cd3-48fc-9336-e2f97e893e40","Type":"ContainerStarted","Data":"37cb1ae19f75ffec343c40410f92f46ab53f1121e89582d8effbe1d569623059"}
Jul 28 12:06:25.882001 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:25.881978    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9636f5f9-tt7j9"
Jul 28 12:06:25.936458 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:25.936432    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ee667f-0cd3-48fc-9336-e2f97e893e40-maas-api-service-ca\") pod \"93ee667f-0cd3-48fc-9336-e2f97e893e40\" (UID: \"93ee667f-0cd3-48fc-9336-e2f97e893e40\") "
Jul 28 12:06:25.936809 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:25.936787    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ee667f-0cd3-48fc-9336-e2f97e893e40-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "93ee667f-0cd3-48fc-9336-e2f97e893e40" (UID: "93ee667f-0cd3-48fc-9336-e2f97e893e40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:06:26.037859 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:26.037801    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ee667f-0cd3-48fc-9336-e2f97e893e40-maas-api-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:06:26.761044 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:26.761015    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9636f5f9-tt7j9" event={"ID":"93ee667f-0cd3-48fc-9336-e2f97e893e40","Type":"ContainerDied","Data":"37cb1ae19f75ffec343c40410f92f46ab53f1121e89582d8effbe1d569623059"}
Jul 28 12:06:26.761044 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:26.761041    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9636f5f9-tt7j9"
Jul 28 12:06:26.761243 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:26.761061    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37cb1ae19f75ffec343c40410f92f46ab53f1121e89582d8effbe1d569623059"
Jul 28 12:06:34.008198 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.008161    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"]
Jul 28 12:06:34.010936 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.008440    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq" podUID="0306a9f3-77bd-4bfd-a635-e62cca900029" containerName="maas-api" containerID="cri-o://641fabe1999467110a065c69d220ea0f86987d6ea3a00b65e809ec7cfc076339" gracePeriod=30
Jul 28 12:06:34.065705 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.065678    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5"]
Jul 28 12:06:34.065928 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.065882    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5" podUID="7ee6d965-ae63-4b8f-add7-b20def1eb982" containerName="payload-processing" containerID="cri-o://3a6aff812a24bffbe9979405064eda2a23994030109d7bf57c2685db987fc7ec" gracePeriod=30
Jul 28 12:06:34.070825 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.070791    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc"]
Jul 28 12:06:34.071009 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.070983    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc" podUID="a687b09b-802d-4b4f-965c-df3d096d01be" containerName="payload-pre-processing" containerID="cri-o://8d3494640294f5ee086f0f6d9a9971a2356f999021814489d7ef4985dd98c24e" gracePeriod=30
Jul 28 12:06:34.339686 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.339663    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"
Jul 28 12:06:34.350575 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.350553    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5"
Jul 28 12:06:34.361307 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.361286    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc"
Jul 28 12:06:34.407421 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.407399    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0306a9f3-77bd-4bfd-a635-e62cca900029-openshift-service-ca\") pod \"0306a9f3-77bd-4bfd-a635-e62cca900029\" (UID: \"0306a9f3-77bd-4bfd-a635-e62cca900029\") "
Jul 28 12:06:34.407530 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.407438    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9rgh\" (UniqueName: \"kubernetes.io/projected/0306a9f3-77bd-4bfd-a635-e62cca900029-kube-api-access-n9rgh\") pod \"0306a9f3-77bd-4bfd-a635-e62cca900029\" (UID: \"0306a9f3-77bd-4bfd-a635-e62cca900029\") "
Jul 28 12:06:34.407530 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.407485    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8wv6\" (UniqueName: \"kubernetes.io/projected/7ee6d965-ae63-4b8f-add7-b20def1eb982-kube-api-access-c8wv6\") pod \"7ee6d965-ae63-4b8f-add7-b20def1eb982\" (UID: \"7ee6d965-ae63-4b8f-add7-b20def1eb982\") "
Jul 28 12:06:34.407530 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.407514    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee6d965-ae63-4b8f-add7-b20def1eb982-plugins-config-volume\") pod \"7ee6d965-ae63-4b8f-add7-b20def1eb982\" (UID: \"7ee6d965-ae63-4b8f-add7-b20def1eb982\") "
Jul 28 12:06:34.407715 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.407603    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0306a9f3-77bd-4bfd-a635-e62cca900029-maas-api-tls\") pod \"0306a9f3-77bd-4bfd-a635-e62cca900029\" (UID: \"0306a9f3-77bd-4bfd-a635-e62cca900029\") "
Jul 28 12:06:34.407877 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.407853    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0306a9f3-77bd-4bfd-a635-e62cca900029-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0306a9f3-77bd-4bfd-a635-e62cca900029" (UID: "0306a9f3-77bd-4bfd-a635-e62cca900029"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:06:34.407982 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.407943    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee6d965-ae63-4b8f-add7-b20def1eb982-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ee6d965-ae63-4b8f-add7-b20def1eb982" (UID: "7ee6d965-ae63-4b8f-add7-b20def1eb982"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:06:34.409650 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.409615    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0306a9f3-77bd-4bfd-a635-e62cca900029-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0306a9f3-77bd-4bfd-a635-e62cca900029" (UID: "0306a9f3-77bd-4bfd-a635-e62cca900029"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:06:34.409832 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.409809    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0306a9f3-77bd-4bfd-a635-e62cca900029-kube-api-access-n9rgh" (OuterVolumeSpecName: "kube-api-access-n9rgh") pod "0306a9f3-77bd-4bfd-a635-e62cca900029" (UID: "0306a9f3-77bd-4bfd-a635-e62cca900029"). InnerVolumeSpecName "kube-api-access-n9rgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:06:34.409948 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.409929    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee6d965-ae63-4b8f-add7-b20def1eb982-kube-api-access-c8wv6" (OuterVolumeSpecName: "kube-api-access-c8wv6") pod "7ee6d965-ae63-4b8f-add7-b20def1eb982" (UID: "7ee6d965-ae63-4b8f-add7-b20def1eb982"). InnerVolumeSpecName "kube-api-access-c8wv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:06:34.508937 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.508909    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6kqc\" (UniqueName: \"kubernetes.io/projected/a687b09b-802d-4b4f-965c-df3d096d01be-kube-api-access-x6kqc\") pod \"a687b09b-802d-4b4f-965c-df3d096d01be\" (UID: \"a687b09b-802d-4b4f-965c-df3d096d01be\") "
Jul 28 12:06:34.509060 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.509018    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a687b09b-802d-4b4f-965c-df3d096d01be-plugins-config-volume\") pod \"a687b09b-802d-4b4f-965c-df3d096d01be\" (UID: \"a687b09b-802d-4b4f-965c-df3d096d01be\") "
Jul 28 12:06:34.509240 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.509223    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9rgh\" (UniqueName: \"kubernetes.io/projected/0306a9f3-77bd-4bfd-a635-e62cca900029-kube-api-access-n9rgh\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:06:34.509304 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.509244    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8wv6\" (UniqueName: \"kubernetes.io/projected/7ee6d965-ae63-4b8f-add7-b20def1eb982-kube-api-access-c8wv6\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:06:34.509304 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.509257    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee6d965-ae63-4b8f-add7-b20def1eb982-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:06:34.509304 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.509270    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0306a9f3-77bd-4bfd-a635-e62cca900029-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:06:34.509304 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.509285    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0306a9f3-77bd-4bfd-a635-e62cca900029-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:06:34.509445 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.509334    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a687b09b-802d-4b4f-965c-df3d096d01be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a687b09b-802d-4b4f-965c-df3d096d01be" (UID: "a687b09b-802d-4b4f-965c-df3d096d01be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:06:34.510662 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.510644    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a687b09b-802d-4b4f-965c-df3d096d01be-kube-api-access-x6kqc" (OuterVolumeSpecName: "kube-api-access-x6kqc") pod "a687b09b-802d-4b4f-965c-df3d096d01be" (UID: "a687b09b-802d-4b4f-965c-df3d096d01be"). InnerVolumeSpecName "kube-api-access-x6kqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:06:34.609914 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.609892    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a687b09b-802d-4b4f-965c-df3d096d01be-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:06:34.609914 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.609913    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6kqc\" (UniqueName: \"kubernetes.io/projected/a687b09b-802d-4b4f-965c-df3d096d01be-kube-api-access-x6kqc\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:06:34.795701 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.795668    2561 generic.go:356] "Generic (PLEG): container finished" podID="7ee6d965-ae63-4b8f-add7-b20def1eb982" containerID="3a6aff812a24bffbe9979405064eda2a23994030109d7bf57c2685db987fc7ec" exitCode=0
Jul 28 12:06:34.795814 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.795713    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5"
Jul 28 12:06:34.795814 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.795747    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5" event={"ID":"7ee6d965-ae63-4b8f-add7-b20def1eb982","Type":"ContainerDied","Data":"3a6aff812a24bffbe9979405064eda2a23994030109d7bf57c2685db987fc7ec"}
Jul 28 12:06:34.795814 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.795792    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5" event={"ID":"7ee6d965-ae63-4b8f-add7-b20def1eb982","Type":"ContainerDied","Data":"5446dddc0b130c14c5fecd9424708f1f8070998fb72c1f1afe58ba39a6ff2d3a"}
Jul 28 12:06:34.795814 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.795814    2561 scope.go:117] "RemoveContainer" containerID="3a6aff812a24bffbe9979405064eda2a23994030109d7bf57c2685db987fc7ec"
Jul 28 12:06:34.797033 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.797011    2561 generic.go:356] "Generic (PLEG): container finished" podID="a687b09b-802d-4b4f-965c-df3d096d01be" containerID="8d3494640294f5ee086f0f6d9a9971a2356f999021814489d7ef4985dd98c24e" exitCode=0
Jul 28 12:06:34.797130 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.797056    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc"
Jul 28 12:06:34.797130 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.797092    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc" event={"ID":"a687b09b-802d-4b4f-965c-df3d096d01be","Type":"ContainerDied","Data":"8d3494640294f5ee086f0f6d9a9971a2356f999021814489d7ef4985dd98c24e"}
Jul 28 12:06:34.797247 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.797153    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc" event={"ID":"a687b09b-802d-4b4f-965c-df3d096d01be","Type":"ContainerDied","Data":"515c6ce0e17f778b0046d32d661d86628b2f1777fd3057d15a55e632395e1c74"}
Jul 28 12:06:34.798771 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.798749    2561 generic.go:356] "Generic (PLEG): container finished" podID="0306a9f3-77bd-4bfd-a635-e62cca900029" containerID="641fabe1999467110a065c69d220ea0f86987d6ea3a00b65e809ec7cfc076339" exitCode=0
Jul 28 12:06:34.798880 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.798787    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"
Jul 28 12:06:34.798880 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.798828    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq" event={"ID":"0306a9f3-77bd-4bfd-a635-e62cca900029","Type":"ContainerDied","Data":"641fabe1999467110a065c69d220ea0f86987d6ea3a00b65e809ec7cfc076339"}
Jul 28 12:06:34.798880 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.798867    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq" event={"ID":"0306a9f3-77bd-4bfd-a635-e62cca900029","Type":"ContainerDied","Data":"994a877e3a17352427cdff1851f143256c79ac60ae274e73b61afacd7420edde"}
Jul 28 12:06:34.804539 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.804516    2561 scope.go:117] "RemoveContainer" containerID="3a6aff812a24bffbe9979405064eda2a23994030109d7bf57c2685db987fc7ec"
Jul 28 12:06:34.804857 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:06:34.804814    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a6aff812a24bffbe9979405064eda2a23994030109d7bf57c2685db987fc7ec\": container with ID starting with 3a6aff812a24bffbe9979405064eda2a23994030109d7bf57c2685db987fc7ec not found: ID does not exist" containerID="3a6aff812a24bffbe9979405064eda2a23994030109d7bf57c2685db987fc7ec"
Jul 28 12:06:34.804857 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.804841    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6aff812a24bffbe9979405064eda2a23994030109d7bf57c2685db987fc7ec"} err="failed to get container status \"3a6aff812a24bffbe9979405064eda2a23994030109d7bf57c2685db987fc7ec\": rpc error: code = NotFound desc = could not find container \"3a6aff812a24bffbe9979405064eda2a23994030109d7bf57c2685db987fc7ec\": container with ID starting with 3a6aff812a24bffbe9979405064eda2a23994030109d7bf57c2685db987fc7ec not found: ID does not exist"
Jul 28 12:06:34.804960 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.804858    2561 scope.go:117] "RemoveContainer" containerID="8d3494640294f5ee086f0f6d9a9971a2356f999021814489d7ef4985dd98c24e"
Jul 28 12:06:34.811563 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.811547    2561 scope.go:117] "RemoveContainer" containerID="8d3494640294f5ee086f0f6d9a9971a2356f999021814489d7ef4985dd98c24e"
Jul 28 12:06:34.811831 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:06:34.811812    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d3494640294f5ee086f0f6d9a9971a2356f999021814489d7ef4985dd98c24e\": container with ID starting with 8d3494640294f5ee086f0f6d9a9971a2356f999021814489d7ef4985dd98c24e not found: ID does not exist" containerID="8d3494640294f5ee086f0f6d9a9971a2356f999021814489d7ef4985dd98c24e"
Jul 28 12:06:34.811906 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.811849    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d3494640294f5ee086f0f6d9a9971a2356f999021814489d7ef4985dd98c24e"} err="failed to get container status \"8d3494640294f5ee086f0f6d9a9971a2356f999021814489d7ef4985dd98c24e\": rpc error: code = NotFound desc = could not find container \"8d3494640294f5ee086f0f6d9a9971a2356f999021814489d7ef4985dd98c24e\": container with ID starting with 8d3494640294f5ee086f0f6d9a9971a2356f999021814489d7ef4985dd98c24e not found: ID does not exist"
Jul 28 12:06:34.811906 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.811874    2561 scope.go:117] "RemoveContainer" containerID="641fabe1999467110a065c69d220ea0f86987d6ea3a00b65e809ec7cfc076339"
Jul 28 12:06:34.814522 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.814504    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5"]
Jul 28 12:06:34.818192 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.818175    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9636f5f9-6bb55b9db6-84lg5"]
Jul 28 12:06:34.820021 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.819995    2561 scope.go:117] "RemoveContainer" containerID="641fabe1999467110a065c69d220ea0f86987d6ea3a00b65e809ec7cfc076339"
Jul 28 12:06:34.820261 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:06:34.820245    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"641fabe1999467110a065c69d220ea0f86987d6ea3a00b65e809ec7cfc076339\": container with ID starting with 641fabe1999467110a065c69d220ea0f86987d6ea3a00b65e809ec7cfc076339 not found: ID does not exist" containerID="641fabe1999467110a065c69d220ea0f86987d6ea3a00b65e809ec7cfc076339"
Jul 28 12:06:34.820354 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.820266    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"641fabe1999467110a065c69d220ea0f86987d6ea3a00b65e809ec7cfc076339"} err="failed to get container status \"641fabe1999467110a065c69d220ea0f86987d6ea3a00b65e809ec7cfc076339\": rpc error: code = NotFound desc = could not find container \"641fabe1999467110a065c69d220ea0f86987d6ea3a00b65e809ec7cfc076339\": container with ID starting with 641fabe1999467110a065c69d220ea0f86987d6ea3a00b65e809ec7cfc076339 not found: ID does not exist"
Jul 28 12:06:34.829499 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.829480    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"]
Jul 28 12:06:34.831642 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.831624    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9636f5f9-7c6cbd679-4mxqq"]
Jul 28 12:06:34.842852 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.842830    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc"]
Jul 28 12:06:34.847018 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:34.847000    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9636f5f9-8954d7fd4-xcmrc"]
Jul 28 12:06:36.546096 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:36.546064    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0306a9f3-77bd-4bfd-a635-e62cca900029" path="/var/lib/kubelet/pods/0306a9f3-77bd-4bfd-a635-e62cca900029/volumes"
Jul 28 12:06:36.546468 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:36.546391    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ee6d965-ae63-4b8f-add7-b20def1eb982" path="/var/lib/kubelet/pods/7ee6d965-ae63-4b8f-add7-b20def1eb982/volumes"
Jul 28 12:06:36.546709 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:36.546697    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a687b09b-802d-4b4f-965c-df3d096d01be" path="/var/lib/kubelet/pods/a687b09b-802d-4b4f-965c-df3d096d01be/volumes"
Jul 28 12:06:37.131070 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:37.131035    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:06:38.660634 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:38.660590    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9636f5f9-tt7j9"]
Jul 28 12:06:38.662336 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:38.662312    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9636f5f9-tt7j9"]
Jul 28 12:06:40.546566 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:06:40.546489    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93ee667f-0cd3-48fc-9336-e2f97e893e40" path="/var/lib/kubelet/pods/93ee667f-0cd3-48fc-9336-e2f97e893e40/volumes"
Jul 28 12:08:06.531876 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:06.531836    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:08:21.328321 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:21.328233    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:08:30.524910 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:30.524876    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:08:36.300521 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.300491    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"]
Jul 28 12:08:36.300894 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.300790    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0306a9f3-77bd-4bfd-a635-e62cca900029" containerName="maas-api"
Jul 28 12:08:36.300894 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.300802    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0306a9f3-77bd-4bfd-a635-e62cca900029" containerName="maas-api"
Jul 28 12:08:36.300894 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.300812    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93ee667f-0cd3-48fc-9336-e2f97e893e40" containerName="revoke-keys"
Jul 28 12:08:36.300894 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.300818    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="93ee667f-0cd3-48fc-9336-e2f97e893e40" containerName="revoke-keys"
Jul 28 12:08:36.300894 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.300832    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee6d965-ae63-4b8f-add7-b20def1eb982" containerName="payload-processing"
Jul 28 12:08:36.300894 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.300838    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee6d965-ae63-4b8f-add7-b20def1eb982" containerName="payload-processing"
Jul 28 12:08:36.300894 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.300850    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a687b09b-802d-4b4f-965c-df3d096d01be" containerName="payload-pre-processing"
Jul 28 12:08:36.300894 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.300855    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a687b09b-802d-4b4f-965c-df3d096d01be" containerName="payload-pre-processing"
Jul 28 12:08:36.300894 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.300899    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="93ee667f-0cd3-48fc-9336-e2f97e893e40" containerName="revoke-keys"
Jul 28 12:08:36.301166 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.300908    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0306a9f3-77bd-4bfd-a635-e62cca900029" containerName="maas-api"
Jul 28 12:08:36.301166 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.300916    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a687b09b-802d-4b4f-965c-df3d096d01be" containerName="payload-pre-processing"
Jul 28 12:08:36.301166 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.300923    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee6d965-ae63-4b8f-add7-b20def1eb982" containerName="payload-processing"
Jul 28 12:08:36.304136 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.304116    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"
Jul 28 12:08:36.307905 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.307883    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9055f75e-serving-cert\""
Jul 28 12:08:36.308357 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.308341    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 12:08:36.308511 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.308495    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xchcw\""
Jul 28 12:08:36.308600 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.308511    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 12:08:36.313510 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.313486    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"]
Jul 28 12:08:36.370685 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.370660    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwhcq\" (UniqueName: \"kubernetes.io/projected/d6a76dfe-11f0-4efd-8974-0743807f2f68-kube-api-access-cwhcq\") pod \"maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r\" (UID: \"d6a76dfe-11f0-4efd-8974-0743807f2f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"
Jul 28 12:08:36.370795 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.370699    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6a76dfe-11f0-4efd-8974-0743807f2f68-openshift-service-ca\") pod \"maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r\" (UID: \"d6a76dfe-11f0-4efd-8974-0743807f2f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"
Jul 28 12:08:36.370795 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.370761    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6a76dfe-11f0-4efd-8974-0743807f2f68-maas-api-tls\") pod \"maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r\" (UID: \"d6a76dfe-11f0-4efd-8974-0743807f2f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"
Jul 28 12:08:36.471778 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.471752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6a76dfe-11f0-4efd-8974-0743807f2f68-maas-api-tls\") pod \"maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r\" (UID: \"d6a76dfe-11f0-4efd-8974-0743807f2f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"
Jul 28 12:08:36.471894 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.471822    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhcq\" (UniqueName: \"kubernetes.io/projected/d6a76dfe-11f0-4efd-8974-0743807f2f68-kube-api-access-cwhcq\") pod \"maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r\" (UID: \"d6a76dfe-11f0-4efd-8974-0743807f2f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"
Jul 28 12:08:36.471894 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.471870    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6a76dfe-11f0-4efd-8974-0743807f2f68-openshift-service-ca\") pod \"maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r\" (UID: \"d6a76dfe-11f0-4efd-8974-0743807f2f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"
Jul 28 12:08:36.472060 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:08:36.471920    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-serving-cert: secret "maas-api-e2e-mt-9055f75e-serving-cert" not found
Jul 28 12:08:36.472060 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:08:36.471998    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6a76dfe-11f0-4efd-8974-0743807f2f68-maas-api-tls podName:d6a76dfe-11f0-4efd-8974-0743807f2f68 nodeName:}" failed. No retries permitted until 2026-07-28 12:08:36.971975648 +0000 UTC m=+2268.830631641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d6a76dfe-11f0-4efd-8974-0743807f2f68-maas-api-tls") pod "maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r" (UID: "d6a76dfe-11f0-4efd-8974-0743807f2f68") : secret "maas-api-e2e-mt-9055f75e-serving-cert" not found
Jul 28 12:08:36.472493 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.472472    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6a76dfe-11f0-4efd-8974-0743807f2f68-openshift-service-ca\") pod \"maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r\" (UID: \"d6a76dfe-11f0-4efd-8974-0743807f2f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"
Jul 28 12:08:36.479998 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.479978    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwhcq\" (UniqueName: \"kubernetes.io/projected/d6a76dfe-11f0-4efd-8974-0743807f2f68-kube-api-access-cwhcq\") pod \"maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r\" (UID: \"d6a76dfe-11f0-4efd-8974-0743807f2f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"
Jul 28 12:08:36.975989 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.975951    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6a76dfe-11f0-4efd-8974-0743807f2f68-maas-api-tls\") pod \"maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r\" (UID: \"d6a76dfe-11f0-4efd-8974-0743807f2f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"
Jul 28 12:08:36.978343 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:36.978326    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6a76dfe-11f0-4efd-8974-0743807f2f68-maas-api-tls\") pod \"maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r\" (UID: \"d6a76dfe-11f0-4efd-8974-0743807f2f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"
Jul 28 12:08:37.214266 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.214221    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"
Jul 28 12:08:37.282495 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.282437    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6"]
Jul 28 12:08:37.286650 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.286629    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6"
Jul 28 12:08:37.295469 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.295449    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9055f75e\""
Jul 28 12:08:37.296205 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.296187    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 12:08:37.296275 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.296263    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9055f75e-dockercfg-8sw84\""
Jul 28 12:08:37.296348 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.296260    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 12:08:37.302957 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.302913    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6"]
Jul 28 12:08:37.330131 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.330104    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75"]
Jul 28 12:08:37.337816 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.337791    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75"
Jul 28 12:08:37.357616 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.357593    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75"]
Jul 28 12:08:37.380083 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.380060    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gps9g\" (UniqueName: \"kubernetes.io/projected/2b785fc1-ccf3-43a3-97ac-77b965e083f7-kube-api-access-gps9g\") pod \"payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6\" (UID: \"2b785fc1-ccf3-43a3-97ac-77b965e083f7\") " pod="openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6"
Jul 28 12:08:37.380195 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.380130    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b785fc1-ccf3-43a3-97ac-77b965e083f7-plugins-config-volume\") pod \"payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6\" (UID: \"2b785fc1-ccf3-43a3-97ac-77b965e083f7\") " pod="openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6"
Jul 28 12:08:37.481500 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.481467    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43fccb9c-68db-474d-b133-3cd5a75b5603-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75\" (UID: \"43fccb9c-68db-474d-b133-3cd5a75b5603\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75"
Jul 28 12:08:37.481676 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.481513    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8br9r\" (UniqueName: \"kubernetes.io/projected/43fccb9c-68db-474d-b133-3cd5a75b5603-kube-api-access-8br9r\") pod \"payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75\" (UID: \"43fccb9c-68db-474d-b133-3cd5a75b5603\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75"
Jul 28 12:08:37.481676 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.481629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b785fc1-ccf3-43a3-97ac-77b965e083f7-plugins-config-volume\") pod \"payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6\" (UID: \"2b785fc1-ccf3-43a3-97ac-77b965e083f7\") " pod="openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6"
Jul 28 12:08:37.481793 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.481680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gps9g\" (UniqueName: \"kubernetes.io/projected/2b785fc1-ccf3-43a3-97ac-77b965e083f7-kube-api-access-gps9g\") pod \"payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6\" (UID: \"2b785fc1-ccf3-43a3-97ac-77b965e083f7\") " pod="openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6"
Jul 28 12:08:37.482302 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.482280    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b785fc1-ccf3-43a3-97ac-77b965e083f7-plugins-config-volume\") pod \"payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6\" (UID: \"2b785fc1-ccf3-43a3-97ac-77b965e083f7\") " pod="openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6"
Jul 28 12:08:37.490390 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.490367    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gps9g\" (UniqueName: \"kubernetes.io/projected/2b785fc1-ccf3-43a3-97ac-77b965e083f7-kube-api-access-gps9g\") pod \"payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6\" (UID: \"2b785fc1-ccf3-43a3-97ac-77b965e083f7\") " pod="openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6"
Jul 28 12:08:37.554655 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.554631    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"]
Jul 28 12:08:37.557099 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:08:37.557076    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a76dfe_11f0_4efd_8974_0743807f2f68.slice/crio-472894292a716e9361c7f0cb7d3436b6eb61cf1a7ec1d795071ea9945252fe81 WatchSource:0}: Error finding container 472894292a716e9361c7f0cb7d3436b6eb61cf1a7ec1d795071ea9945252fe81: Status 404 returned error can't find the container with id 472894292a716e9361c7f0cb7d3436b6eb61cf1a7ec1d795071ea9945252fe81
Jul 28 12:08:37.582332 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.582308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43fccb9c-68db-474d-b133-3cd5a75b5603-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75\" (UID: \"43fccb9c-68db-474d-b133-3cd5a75b5603\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75"
Jul 28 12:08:37.582416 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.582348    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8br9r\" (UniqueName: \"kubernetes.io/projected/43fccb9c-68db-474d-b133-3cd5a75b5603-kube-api-access-8br9r\") pod \"payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75\" (UID: \"43fccb9c-68db-474d-b133-3cd5a75b5603\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75"
Jul 28 12:08:37.582991 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.582972    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43fccb9c-68db-474d-b133-3cd5a75b5603-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75\" (UID: \"43fccb9c-68db-474d-b133-3cd5a75b5603\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75"
Jul 28 12:08:37.590334 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.590310    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8br9r\" (UniqueName: \"kubernetes.io/projected/43fccb9c-68db-474d-b133-3cd5a75b5603-kube-api-access-8br9r\") pod \"payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75\" (UID: \"43fccb9c-68db-474d-b133-3cd5a75b5603\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75"
Jul 28 12:08:37.597992 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.597973    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6"
Jul 28 12:08:37.648010 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.647956    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75"
Jul 28 12:08:37.733553 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.733524    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6"]
Jul 28 12:08:37.735848 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:08:37.735820    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b785fc1_ccf3_43a3_97ac_77b965e083f7.slice/crio-7be5c3476249e26b1dbd63086a6b26686badc2c23adf011fc7ca351ed4ca996a WatchSource:0}: Error finding container 7be5c3476249e26b1dbd63086a6b26686badc2c23adf011fc7ca351ed4ca996a: Status 404 returned error can't find the container with id 7be5c3476249e26b1dbd63086a6b26686badc2c23adf011fc7ca351ed4ca996a
Jul 28 12:08:37.789256 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:37.789189    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75"]
Jul 28 12:08:37.791628 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:08:37.791605    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43fccb9c_68db_474d_b133_3cd5a75b5603.slice/crio-ef32d6ba0e01608b7aedf82bb77c0217ab76fa779a27d75cd5101bdcc7ee078b WatchSource:0}: Error finding container ef32d6ba0e01608b7aedf82bb77c0217ab76fa779a27d75cd5101bdcc7ee078b: Status 404 returned error can't find the container with id ef32d6ba0e01608b7aedf82bb77c0217ab76fa779a27d75cd5101bdcc7ee078b
Jul 28 12:08:38.330825 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:38.330785    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6" event={"ID":"2b785fc1-ccf3-43a3-97ac-77b965e083f7","Type":"ContainerStarted","Data":"79d0d5cbc19e066934d91b3067fd57fcec3fbc6aa2ef90d7533b100406bd772d"}
Jul 28 12:08:38.331271 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:38.330848    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6" event={"ID":"2b785fc1-ccf3-43a3-97ac-77b965e083f7","Type":"ContainerStarted","Data":"7be5c3476249e26b1dbd63086a6b26686badc2c23adf011fc7ca351ed4ca996a"}
Jul 28 12:08:38.331271 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:38.330929    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6"
Jul 28 12:08:38.331915 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:38.331893    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r" event={"ID":"d6a76dfe-11f0-4efd-8974-0743807f2f68","Type":"ContainerStarted","Data":"472894292a716e9361c7f0cb7d3436b6eb61cf1a7ec1d795071ea9945252fe81"}
Jul 28 12:08:38.333419 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:38.333390    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75" event={"ID":"43fccb9c-68db-474d-b133-3cd5a75b5603","Type":"ContainerStarted","Data":"5450f4045fdd7a0d925e316f796c8999cc0f474fc5ee91ee6363816a47816e64"}
Jul 28 12:08:38.333523 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:38.333435    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75" event={"ID":"43fccb9c-68db-474d-b133-3cd5a75b5603","Type":"ContainerStarted","Data":"ef32d6ba0e01608b7aedf82bb77c0217ab76fa779a27d75cd5101bdcc7ee078b"}
Jul 28 12:08:38.333566 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:38.333525    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75"
Jul 28 12:08:38.347321 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:38.347279    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6" podStartSLOduration=1.347266028 podStartE2EDuration="1.347266028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:08:38.346018377 +0000 UTC m=+2270.204674378" watchObservedRunningTime="2026-07-28 12:08:38.347266028 +0000 UTC m=+2270.205922027"
Jul 28 12:08:38.361829 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:38.361779    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75" podStartSLOduration=1.36176676 podStartE2EDuration="1.36176676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:08:38.361064184 +0000 UTC m=+2270.219720184" watchObservedRunningTime="2026-07-28 12:08:38.36176676 +0000 UTC m=+2270.220422760"
Jul 28 12:08:39.339342 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:39.339305    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r" event={"ID":"d6a76dfe-11f0-4efd-8974-0743807f2f68","Type":"ContainerStarted","Data":"e4bccca620ef395d337c4d7aa1276fa74e8becfb2630176a2b2a07694eaac805"}
Jul 28 12:08:39.339799 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:39.339635    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"
Jul 28 12:08:39.359606 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:39.359548    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r" podStartSLOduration=2.064367613 podStartE2EDuration="3.359535691s" totalImagesPullingTime="1.295168078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:08:39.358664867 +0000 UTC m=+2271.217320869" watchObservedRunningTime="2026-07-28 12:08:39.359535691 +0000 UTC m=+2271.218191690"
Jul 28 12:08:41.840273 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:41.840240    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:08:45.350131 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:45.350101    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"
Jul 28 12:08:49.341281 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:49.341249    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75"
Jul 28 12:08:49.341733 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:49.341433    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6"
Jul 28 12:08:58.909655 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:58.909623    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"]
Jul 28 12:08:58.910294 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:58.909865    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r" podUID="d6a76dfe-11f0-4efd-8974-0743807f2f68" containerName="maas-api" containerID="cri-o://e4bccca620ef395d337c4d7aa1276fa74e8becfb2630176a2b2a07694eaac805" gracePeriod=30
Jul 28 12:08:58.970071 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:58.970026    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6"]
Jul 28 12:08:58.970308 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:58.970258    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6" podUID="2b785fc1-ccf3-43a3-97ac-77b965e083f7" containerName="payload-processing" containerID="cri-o://79d0d5cbc19e066934d91b3067fd57fcec3fbc6aa2ef90d7533b100406bd772d" gracePeriod=30
Jul 28 12:08:58.996448 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:58.996408    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75"]
Jul 28 12:08:58.996784 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:58.996750    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75" podUID="43fccb9c-68db-474d-b133-3cd5a75b5603" containerName="payload-pre-processing" containerID="cri-o://5450f4045fdd7a0d925e316f796c8999cc0f474fc5ee91ee6363816a47816e64" gracePeriod=30
Jul 28 12:08:59.254416 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.254392    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"
Jul 28 12:08:59.283332 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.283308    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75"
Jul 28 12:08:59.290720 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.290699    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6"
Jul 28 12:08:59.368641 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.368613    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6a76dfe-11f0-4efd-8974-0743807f2f68-maas-api-tls\") pod \"d6a76dfe-11f0-4efd-8974-0743807f2f68\" (UID: \"d6a76dfe-11f0-4efd-8974-0743807f2f68\") "
Jul 28 12:08:59.368829 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.368709    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwhcq\" (UniqueName: \"kubernetes.io/projected/d6a76dfe-11f0-4efd-8974-0743807f2f68-kube-api-access-cwhcq\") pod \"d6a76dfe-11f0-4efd-8974-0743807f2f68\" (UID: \"d6a76dfe-11f0-4efd-8974-0743807f2f68\") "
Jul 28 12:08:59.368829 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.368777    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6a76dfe-11f0-4efd-8974-0743807f2f68-openshift-service-ca\") pod \"d6a76dfe-11f0-4efd-8974-0743807f2f68\" (UID: \"d6a76dfe-11f0-4efd-8974-0743807f2f68\") "
Jul 28 12:08:59.369118 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.369095    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a76dfe-11f0-4efd-8974-0743807f2f68-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d6a76dfe-11f0-4efd-8974-0743807f2f68" (UID: "d6a76dfe-11f0-4efd-8974-0743807f2f68"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:08:59.370779 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.370758    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a76dfe-11f0-4efd-8974-0743807f2f68-kube-api-access-cwhcq" (OuterVolumeSpecName: "kube-api-access-cwhcq") pod "d6a76dfe-11f0-4efd-8974-0743807f2f68" (UID: "d6a76dfe-11f0-4efd-8974-0743807f2f68"). InnerVolumeSpecName "kube-api-access-cwhcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:08:59.370897 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.370877    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6a76dfe-11f0-4efd-8974-0743807f2f68-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d6a76dfe-11f0-4efd-8974-0743807f2f68" (UID: "d6a76dfe-11f0-4efd-8974-0743807f2f68"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:08:59.426610 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.426507    2561 generic.go:356] "Generic (PLEG): container finished" podID="2b785fc1-ccf3-43a3-97ac-77b965e083f7" containerID="79d0d5cbc19e066934d91b3067fd57fcec3fbc6aa2ef90d7533b100406bd772d" exitCode=0
Jul 28 12:08:59.426610 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.426550    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6"
Jul 28 12:08:59.426762 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.426615    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6" event={"ID":"2b785fc1-ccf3-43a3-97ac-77b965e083f7","Type":"ContainerDied","Data":"79d0d5cbc19e066934d91b3067fd57fcec3fbc6aa2ef90d7533b100406bd772d"}
Jul 28 12:08:59.426762 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.426666    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6" event={"ID":"2b785fc1-ccf3-43a3-97ac-77b965e083f7","Type":"ContainerDied","Data":"7be5c3476249e26b1dbd63086a6b26686badc2c23adf011fc7ca351ed4ca996a"}
Jul 28 12:08:59.426762 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.426692    2561 scope.go:117] "RemoveContainer" containerID="79d0d5cbc19e066934d91b3067fd57fcec3fbc6aa2ef90d7533b100406bd772d"
Jul 28 12:08:59.427759 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.427737    2561 generic.go:356] "Generic (PLEG): container finished" podID="d6a76dfe-11f0-4efd-8974-0743807f2f68" containerID="e4bccca620ef395d337c4d7aa1276fa74e8becfb2630176a2b2a07694eaac805" exitCode=0
Jul 28 12:08:59.427865 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.427763    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r" event={"ID":"d6a76dfe-11f0-4efd-8974-0743807f2f68","Type":"ContainerDied","Data":"e4bccca620ef395d337c4d7aa1276fa74e8becfb2630176a2b2a07694eaac805"}
Jul 28 12:08:59.427865 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.427778    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"
Jul 28 12:08:59.427865 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.427803    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r" event={"ID":"d6a76dfe-11f0-4efd-8974-0743807f2f68","Type":"ContainerDied","Data":"472894292a716e9361c7f0cb7d3436b6eb61cf1a7ec1d795071ea9945252fe81"}
Jul 28 12:08:59.429249 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.429225    2561 generic.go:356] "Generic (PLEG): container finished" podID="43fccb9c-68db-474d-b133-3cd5a75b5603" containerID="5450f4045fdd7a0d925e316f796c8999cc0f474fc5ee91ee6363816a47816e64" exitCode=0
Jul 28 12:08:59.429351 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.429256    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75" event={"ID":"43fccb9c-68db-474d-b133-3cd5a75b5603","Type":"ContainerDied","Data":"5450f4045fdd7a0d925e316f796c8999cc0f474fc5ee91ee6363816a47816e64"}
Jul 28 12:08:59.429351 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.429295    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75" event={"ID":"43fccb9c-68db-474d-b133-3cd5a75b5603","Type":"ContainerDied","Data":"ef32d6ba0e01608b7aedf82bb77c0217ab76fa779a27d75cd5101bdcc7ee078b"}
Jul 28 12:08:59.429351 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.429263    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75"
Jul 28 12:08:59.435943 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.435924    2561 scope.go:117] "RemoveContainer" containerID="79d0d5cbc19e066934d91b3067fd57fcec3fbc6aa2ef90d7533b100406bd772d"
Jul 28 12:08:59.436203 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:08:59.436187    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d0d5cbc19e066934d91b3067fd57fcec3fbc6aa2ef90d7533b100406bd772d\": container with ID starting with 79d0d5cbc19e066934d91b3067fd57fcec3fbc6aa2ef90d7533b100406bd772d not found: ID does not exist" containerID="79d0d5cbc19e066934d91b3067fd57fcec3fbc6aa2ef90d7533b100406bd772d"
Jul 28 12:08:59.436270 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.436211    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d0d5cbc19e066934d91b3067fd57fcec3fbc6aa2ef90d7533b100406bd772d"} err="failed to get container status \"79d0d5cbc19e066934d91b3067fd57fcec3fbc6aa2ef90d7533b100406bd772d\": rpc error: code = NotFound desc = could not find container \"79d0d5cbc19e066934d91b3067fd57fcec3fbc6aa2ef90d7533b100406bd772d\": container with ID starting with 79d0d5cbc19e066934d91b3067fd57fcec3fbc6aa2ef90d7533b100406bd772d not found: ID does not exist"
Jul 28 12:08:59.436270 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.436232    2561 scope.go:117] "RemoveContainer" containerID="e4bccca620ef395d337c4d7aa1276fa74e8becfb2630176a2b2a07694eaac805"
Jul 28 12:08:59.443762 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.443742    2561 scope.go:117] "RemoveContainer" containerID="e4bccca620ef395d337c4d7aa1276fa74e8becfb2630176a2b2a07694eaac805"
Jul 28 12:08:59.443980 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:08:59.443964    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4bccca620ef395d337c4d7aa1276fa74e8becfb2630176a2b2a07694eaac805\": container with ID starting with e4bccca620ef395d337c4d7aa1276fa74e8becfb2630176a2b2a07694eaac805 not found: ID does not exist" containerID="e4bccca620ef395d337c4d7aa1276fa74e8becfb2630176a2b2a07694eaac805"
Jul 28 12:08:59.444030 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.443984    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4bccca620ef395d337c4d7aa1276fa74e8becfb2630176a2b2a07694eaac805"} err="failed to get container status \"e4bccca620ef395d337c4d7aa1276fa74e8becfb2630176a2b2a07694eaac805\": rpc error: code = NotFound desc = could not find container \"e4bccca620ef395d337c4d7aa1276fa74e8becfb2630176a2b2a07694eaac805\": container with ID starting with e4bccca620ef395d337c4d7aa1276fa74e8becfb2630176a2b2a07694eaac805 not found: ID does not exist"
Jul 28 12:08:59.444030 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.443999    2561 scope.go:117] "RemoveContainer" containerID="5450f4045fdd7a0d925e316f796c8999cc0f474fc5ee91ee6363816a47816e64"
Jul 28 12:08:59.449132 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.449113    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"]
Jul 28 12:08:59.451019 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.451005    2561 scope.go:117] "RemoveContainer" containerID="5450f4045fdd7a0d925e316f796c8999cc0f474fc5ee91ee6363816a47816e64"
Jul 28 12:08:59.451273 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:08:59.451253    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5450f4045fdd7a0d925e316f796c8999cc0f474fc5ee91ee6363816a47816e64\": container with ID starting with 5450f4045fdd7a0d925e316f796c8999cc0f474fc5ee91ee6363816a47816e64 not found: ID does not exist" containerID="5450f4045fdd7a0d925e316f796c8999cc0f474fc5ee91ee6363816a47816e64"
Jul 28 12:08:59.451336 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.451278    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5450f4045fdd7a0d925e316f796c8999cc0f474fc5ee91ee6363816a47816e64"} err="failed to get container status \"5450f4045fdd7a0d925e316f796c8999cc0f474fc5ee91ee6363816a47816e64\": rpc error: code = NotFound desc = could not find container \"5450f4045fdd7a0d925e316f796c8999cc0f474fc5ee91ee6363816a47816e64\": container with ID starting with 5450f4045fdd7a0d925e316f796c8999cc0f474fc5ee91ee6363816a47816e64 not found: ID does not exist"
Jul 28 12:08:59.455244 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.455224    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9055f75e-b59c4c48f-2md9r"]
Jul 28 12:08:59.470427 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.470408    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43fccb9c-68db-474d-b133-3cd5a75b5603-plugins-config-volume\") pod \"43fccb9c-68db-474d-b133-3cd5a75b5603\" (UID: \"43fccb9c-68db-474d-b133-3cd5a75b5603\") "
Jul 28 12:08:59.470684 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.470655    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gps9g\" (UniqueName: \"kubernetes.io/projected/2b785fc1-ccf3-43a3-97ac-77b965e083f7-kube-api-access-gps9g\") pod \"2b785fc1-ccf3-43a3-97ac-77b965e083f7\" (UID: \"2b785fc1-ccf3-43a3-97ac-77b965e083f7\") "
Jul 28 12:08:59.471825 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.471136    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8br9r\" (UniqueName: \"kubernetes.io/projected/43fccb9c-68db-474d-b133-3cd5a75b5603-kube-api-access-8br9r\") pod \"43fccb9c-68db-474d-b133-3cd5a75b5603\" (UID: \"43fccb9c-68db-474d-b133-3cd5a75b5603\") "
Jul 28 12:08:59.471825 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.471234    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b785fc1-ccf3-43a3-97ac-77b965e083f7-plugins-config-volume\") pod \"2b785fc1-ccf3-43a3-97ac-77b965e083f7\" (UID: \"2b785fc1-ccf3-43a3-97ac-77b965e083f7\") "
Jul 28 12:08:59.471825 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.471608    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6a76dfe-11f0-4efd-8974-0743807f2f68-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:08:59.471825 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.471626    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwhcq\" (UniqueName: \"kubernetes.io/projected/d6a76dfe-11f0-4efd-8974-0743807f2f68-kube-api-access-cwhcq\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:08:59.471825 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.471646    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6a76dfe-11f0-4efd-8974-0743807f2f68-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:08:59.471825 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.471664    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43fccb9c-68db-474d-b133-3cd5a75b5603-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "43fccb9c-68db-474d-b133-3cd5a75b5603" (UID: "43fccb9c-68db-474d-b133-3cd5a75b5603"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:08:59.473673 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.472241    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b785fc1-ccf3-43a3-97ac-77b965e083f7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2b785fc1-ccf3-43a3-97ac-77b965e083f7" (UID: "2b785fc1-ccf3-43a3-97ac-77b965e083f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:08:59.475864 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.475838    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b785fc1-ccf3-43a3-97ac-77b965e083f7-kube-api-access-gps9g" (OuterVolumeSpecName: "kube-api-access-gps9g") pod "2b785fc1-ccf3-43a3-97ac-77b965e083f7" (UID: "2b785fc1-ccf3-43a3-97ac-77b965e083f7"). InnerVolumeSpecName "kube-api-access-gps9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:08:59.477381 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.477356    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43fccb9c-68db-474d-b133-3cd5a75b5603-kube-api-access-8br9r" (OuterVolumeSpecName: "kube-api-access-8br9r") pod "43fccb9c-68db-474d-b133-3cd5a75b5603" (UID: "43fccb9c-68db-474d-b133-3cd5a75b5603"). InnerVolumeSpecName "kube-api-access-8br9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:08:59.572449 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.572421    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b785fc1-ccf3-43a3-97ac-77b965e083f7-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:08:59.572449 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.572445    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43fccb9c-68db-474d-b133-3cd5a75b5603-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:08:59.572449 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.572454    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gps9g\" (UniqueName: \"kubernetes.io/projected/2b785fc1-ccf3-43a3-97ac-77b965e083f7-kube-api-access-gps9g\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:08:59.572699 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.572463    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8br9r\" (UniqueName: \"kubernetes.io/projected/43fccb9c-68db-474d-b133-3cd5a75b5603-kube-api-access-8br9r\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:08:59.748744 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.748715    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6"]
Jul 28 12:08:59.753218 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.753194    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9055f75e-758d548f7d-wkfs6"]
Jul 28 12:08:59.762615 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.762594    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75"]
Jul 28 12:08:59.769107 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:08:59.769084    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9055f75e-5d7cb8757b-wvk75"]
Jul 28 12:09:00.546227 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:00.546193    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b785fc1-ccf3-43a3-97ac-77b965e083f7" path="/var/lib/kubelet/pods/2b785fc1-ccf3-43a3-97ac-77b965e083f7/volumes"
Jul 28 12:09:00.546620 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:00.546534    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43fccb9c-68db-474d-b133-3cd5a75b5603" path="/var/lib/kubelet/pods/43fccb9c-68db-474d-b133-3cd5a75b5603/volumes"
Jul 28 12:09:00.546893 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:00.546880    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6a76dfe-11f0-4efd-8974-0743807f2f68" path="/var/lib/kubelet/pods/d6a76dfe-11f0-4efd-8974-0743807f2f68/volumes"
Jul 28 12:09:02.036322 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:02.036290    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:09:35.527523 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.527492    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"]
Jul 28 12:09:35.528011 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.527805    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b785fc1-ccf3-43a3-97ac-77b965e083f7" containerName="payload-processing"
Jul 28 12:09:35.528011 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.527818    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b785fc1-ccf3-43a3-97ac-77b965e083f7" containerName="payload-processing"
Jul 28 12:09:35.528011 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.527827    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6a76dfe-11f0-4efd-8974-0743807f2f68" containerName="maas-api"
Jul 28 12:09:35.528011 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.527832    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6a76dfe-11f0-4efd-8974-0743807f2f68" containerName="maas-api"
Jul 28 12:09:35.528011 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.527844    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43fccb9c-68db-474d-b133-3cd5a75b5603" containerName="payload-pre-processing"
Jul 28 12:09:35.528011 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.527850    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="43fccb9c-68db-474d-b133-3cd5a75b5603" containerName="payload-pre-processing"
Jul 28 12:09:35.528011 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.527907    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="43fccb9c-68db-474d-b133-3cd5a75b5603" containerName="payload-pre-processing"
Jul 28 12:09:35.528011 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.527915    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b785fc1-ccf3-43a3-97ac-77b965e083f7" containerName="payload-processing"
Jul 28 12:09:35.528011 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.527921    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6a76dfe-11f0-4efd-8974-0743807f2f68" containerName="maas-api"
Jul 28 12:09:35.530973 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.530953    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"
Jul 28 12:09:35.533840 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.533818    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 12:09:35.533936 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.533826    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7038fa7e-serving-cert\""
Jul 28 12:09:35.534462 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.534446    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 12:09:35.534504 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.534475    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xchcw\""
Jul 28 12:09:35.550690 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.550670    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"]
Jul 28 12:09:35.654103 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.654077    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f51d7381-2d43-46c0-b798-96e8c9425869-maas-api-tls\") pod \"maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn\" (UID: \"f51d7381-2d43-46c0-b798-96e8c9425869\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"
Jul 28 12:09:35.654244 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.654107    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51d7381-2d43-46c0-b798-96e8c9425869-openshift-service-ca\") pod \"maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn\" (UID: \"f51d7381-2d43-46c0-b798-96e8c9425869\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"
Jul 28 12:09:35.654244 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.654187    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrcdj\" (UniqueName: \"kubernetes.io/projected/f51d7381-2d43-46c0-b798-96e8c9425869-kube-api-access-mrcdj\") pod \"maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn\" (UID: \"f51d7381-2d43-46c0-b798-96e8c9425869\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"
Jul 28 12:09:35.754977 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.754950    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51d7381-2d43-46c0-b798-96e8c9425869-openshift-service-ca\") pod \"maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn\" (UID: \"f51d7381-2d43-46c0-b798-96e8c9425869\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"
Jul 28 12:09:35.755158 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.755014    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrcdj\" (UniqueName: \"kubernetes.io/projected/f51d7381-2d43-46c0-b798-96e8c9425869-kube-api-access-mrcdj\") pod \"maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn\" (UID: \"f51d7381-2d43-46c0-b798-96e8c9425869\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"
Jul 28 12:09:35.755158 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.755052    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f51d7381-2d43-46c0-b798-96e8c9425869-maas-api-tls\") pod \"maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn\" (UID: \"f51d7381-2d43-46c0-b798-96e8c9425869\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"
Jul 28 12:09:35.755565 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.755539    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51d7381-2d43-46c0-b798-96e8c9425869-openshift-service-ca\") pod \"maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn\" (UID: \"f51d7381-2d43-46c0-b798-96e8c9425869\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"
Jul 28 12:09:35.757277 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.757256    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f51d7381-2d43-46c0-b798-96e8c9425869-maas-api-tls\") pod \"maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn\" (UID: \"f51d7381-2d43-46c0-b798-96e8c9425869\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"
Jul 28 12:09:35.768392 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.768369    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrcdj\" (UniqueName: \"kubernetes.io/projected/f51d7381-2d43-46c0-b798-96e8c9425869-kube-api-access-mrcdj\") pod \"maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn\" (UID: \"f51d7381-2d43-46c0-b798-96e8c9425869\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"
Jul 28 12:09:35.840712 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.840687    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"
Jul 28 12:09:35.977510 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:35.977480    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"]
Jul 28 12:09:35.978619 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:09:35.978570    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51d7381_2d43_46c0_b798_96e8c9425869.slice/crio-4e07ed03f939c82982bab61e28ac425ff7e01f2c7f7e41b7a553b7f479606289 WatchSource:0}: Error finding container 4e07ed03f939c82982bab61e28ac425ff7e01f2c7f7e41b7a553b7f479606289: Status 404 returned error can't find the container with id 4e07ed03f939c82982bab61e28ac425ff7e01f2c7f7e41b7a553b7f479606289
Jul 28 12:09:36.587477 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.587446    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn" event={"ID":"f51d7381-2d43-46c0-b798-96e8c9425869","Type":"ContainerStarted","Data":"f289b83cb8af6cf80c974bea7fff25d4ead6a05482355bcec6aa0d601c0544d8"}
Jul 28 12:09:36.587895 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.587497    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn" event={"ID":"f51d7381-2d43-46c0-b798-96e8c9425869","Type":"ContainerStarted","Data":"4e07ed03f939c82982bab61e28ac425ff7e01f2c7f7e41b7a553b7f479606289"}
Jul 28 12:09:36.587895 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.587553    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"
Jul 28 12:09:36.598752 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.598729    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx"]
Jul 28 12:09:36.601965 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.601949    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx"
Jul 28 12:09:36.605269 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.605248    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 12:09:36.605269 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.605256    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 12:09:36.605870 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.605855    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7038fa7e-dockercfg-8nqxm\""
Jul 28 12:09:36.606466 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.606444    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7038fa7e\""
Jul 28 12:09:36.624759 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.624736    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx"]
Jul 28 12:09:36.625722 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.625678    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn" podStartSLOduration=1.158985786 podStartE2EDuration="1.625666077s" totalImagesPullingTime="466.680291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:09:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:09:36.622238326 +0000 UTC m=+2328.480894341" watchObservedRunningTime="2026-07-28 12:09:36.625666077 +0000 UTC m=+2328.484322075"
Jul 28 12:09:36.661742 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.661718    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcd9b4f6-6b04-4b3f-af07-628352d67ca6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx\" (UID: \"dcd9b4f6-6b04-4b3f-af07-628352d67ca6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx"
Jul 28 12:09:36.661858 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.661825    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8rzq\" (UniqueName: \"kubernetes.io/projected/dcd9b4f6-6b04-4b3f-af07-628352d67ca6-kube-api-access-x8rzq\") pod \"payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx\" (UID: \"dcd9b4f6-6b04-4b3f-af07-628352d67ca6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx"
Jul 28 12:09:36.704170 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.704106    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v"]
Jul 28 12:09:36.707655 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.707640    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v"
Jul 28 12:09:36.725769 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.725746    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v"]
Jul 28 12:09:36.762335 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.762314    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcd9b4f6-6b04-4b3f-af07-628352d67ca6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx\" (UID: \"dcd9b4f6-6b04-4b3f-af07-628352d67ca6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx"
Jul 28 12:09:36.762442 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.762345    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8rzq\" (UniqueName: \"kubernetes.io/projected/dcd9b4f6-6b04-4b3f-af07-628352d67ca6-kube-api-access-x8rzq\") pod \"payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx\" (UID: \"dcd9b4f6-6b04-4b3f-af07-628352d67ca6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx"
Jul 28 12:09:36.762442 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.762371    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knjqb\" (UniqueName: \"kubernetes.io/projected/d2db8331-1573-4770-9798-437f930f131d-kube-api-access-knjqb\") pod \"payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v\" (UID: \"d2db8331-1573-4770-9798-437f930f131d\") " pod="openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v"
Jul 28 12:09:36.762442 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.762393    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2db8331-1573-4770-9798-437f930f131d-plugins-config-volume\") pod \"payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v\" (UID: \"d2db8331-1573-4770-9798-437f930f131d\") " pod="openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v"
Jul 28 12:09:36.762986 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.762962    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcd9b4f6-6b04-4b3f-af07-628352d67ca6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx\" (UID: \"dcd9b4f6-6b04-4b3f-af07-628352d67ca6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx"
Jul 28 12:09:36.772320 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.772294    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8rzq\" (UniqueName: \"kubernetes.io/projected/dcd9b4f6-6b04-4b3f-af07-628352d67ca6-kube-api-access-x8rzq\") pod \"payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx\" (UID: \"dcd9b4f6-6b04-4b3f-af07-628352d67ca6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx"
Jul 28 12:09:36.863435 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.863410    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2db8331-1573-4770-9798-437f930f131d-plugins-config-volume\") pod \"payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v\" (UID: \"d2db8331-1573-4770-9798-437f930f131d\") " pod="openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v"
Jul 28 12:09:36.863544 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.863494    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knjqb\" (UniqueName: \"kubernetes.io/projected/d2db8331-1573-4770-9798-437f930f131d-kube-api-access-knjqb\") pod \"payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v\" (UID: \"d2db8331-1573-4770-9798-437f930f131d\") " pod="openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v"
Jul 28 12:09:36.864067 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.864048    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2db8331-1573-4770-9798-437f930f131d-plugins-config-volume\") pod \"payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v\" (UID: \"d2db8331-1573-4770-9798-437f930f131d\") " pod="openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v"
Jul 28 12:09:36.881186 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.881149    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knjqb\" (UniqueName: \"kubernetes.io/projected/d2db8331-1573-4770-9798-437f930f131d-kube-api-access-knjqb\") pod \"payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v\" (UID: \"d2db8331-1573-4770-9798-437f930f131d\") " pod="openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v"
Jul 28 12:09:36.911735 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:36.911704    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx"
Jul 28 12:09:37.017074 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:37.017048    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v"
Jul 28 12:09:37.057901 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:37.057868    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx"]
Jul 28 12:09:37.060654 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:09:37.060630    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd9b4f6_6b04_4b3f_af07_628352d67ca6.slice/crio-96103ae22e3dd99279b3bf1ebc6ae5d7dfd98dd897e31909f577dd5c950eb840 WatchSource:0}: Error finding container 96103ae22e3dd99279b3bf1ebc6ae5d7dfd98dd897e31909f577dd5c950eb840: Status 404 returned error can't find the container with id 96103ae22e3dd99279b3bf1ebc6ae5d7dfd98dd897e31909f577dd5c950eb840
Jul 28 12:09:37.160626 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:37.160593    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v"]
Jul 28 12:09:37.163250 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:09:37.163226    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2db8331_1573_4770_9798_437f930f131d.slice/crio-2d1d872c7781db91af93e1195f911a4b844eb80b4b620daa67d6a60d32e88361 WatchSource:0}: Error finding container 2d1d872c7781db91af93e1195f911a4b844eb80b4b620daa67d6a60d32e88361: Status 404 returned error can't find the container with id 2d1d872c7781db91af93e1195f911a4b844eb80b4b620daa67d6a60d32e88361
Jul 28 12:09:37.593168 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:37.593126    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx" event={"ID":"dcd9b4f6-6b04-4b3f-af07-628352d67ca6","Type":"ContainerStarted","Data":"298a243e50aee30294cffd0bba75fe785799f1e531522739a103873e595f208f"}
Jul 28 12:09:37.593540 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:37.593213    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx"
Jul 28 12:09:37.593540 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:37.593243    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx" event={"ID":"dcd9b4f6-6b04-4b3f-af07-628352d67ca6","Type":"ContainerStarted","Data":"96103ae22e3dd99279b3bf1ebc6ae5d7dfd98dd897e31909f577dd5c950eb840"}
Jul 28 12:09:37.594951 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:37.594926    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v" event={"ID":"d2db8331-1573-4770-9798-437f930f131d","Type":"ContainerStarted","Data":"51dd7a2a10697265d5afbe3daace95e74596fad13a68de02d4639367adb55b66"}
Jul 28 12:09:37.595049 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:37.594975    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v" event={"ID":"d2db8331-1573-4770-9798-437f930f131d","Type":"ContainerStarted","Data":"2d1d872c7781db91af93e1195f911a4b844eb80b4b620daa67d6a60d32e88361"}
Jul 28 12:09:37.612570 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:37.612524    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx" podStartSLOduration=1.612511893 podStartE2EDuration="1.612511893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:09:37.611013631 +0000 UTC m=+2329.469669627" watchObservedRunningTime="2026-07-28 12:09:37.612511893 +0000 UTC m=+2329.471167891"
Jul 28 12:09:37.632460 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:37.632420    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v" podStartSLOduration=1.632406923 podStartE2EDuration="1.632406923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:09:37.630746403 +0000 UTC m=+2329.489402403" watchObservedRunningTime="2026-07-28 12:09:37.632406923 +0000 UTC m=+2329.491062899"
Jul 28 12:09:38.599275 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:38.599230    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v"
Jul 28 12:09:41.531308 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:41.531225    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:09:42.600291 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:42.600258    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"
Jul 28 12:09:46.302769 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.302732    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"]
Jul 28 12:09:46.307703 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.307684    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:46.309804 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.309783    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7038fa7e\"/\"openshift-service-ca.crt\""
Jul 28 12:09:46.310457 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.310440    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7038fa7e\"/\"default-dockercfg-prjr8\""
Jul 28 12:09:46.310457 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.310451    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7038fa7e\"/\"kube-root-ca.crt\""
Jul 28 12:09:46.310614 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.310480    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7038fa7e\"/\"e2e-lifecycle-model-7038fa7e-kserve-self-signed-certs\""
Jul 28 12:09:46.316606 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.316557    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"]
Jul 28 12:09:46.453947 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.453908    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-home\") pod \"e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") " pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:46.453947 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.453952    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-tmp-dir\") pod \"e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") " pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:46.454162 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.453977    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-dshm\") pod \"e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") " pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:46.454162 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.454044    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89b4886d-bfd4-4468-9db7-cd87445e24aa-tls-certs\") pod \"e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") " pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:46.454162 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.454114    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-model-cache\") pod \"e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") " pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:46.454162 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.454146    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdnfm\" (UniqueName: \"kubernetes.io/projected/89b4886d-bfd4-4468-9db7-cd87445e24aa-kube-api-access-fdnfm\") pod \"e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") " pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:46.554610 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.554518    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-home\") pod \"e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") " pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:46.554610 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.554553    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-tmp-dir\") pod \"e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") " pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:46.554610 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.554571    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-dshm\") pod \"e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") " pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:46.554873 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.554621    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89b4886d-bfd4-4468-9db7-cd87445e24aa-tls-certs\") pod \"e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") " pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:46.554873 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.554664    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-model-cache\") pod \"e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") " pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:46.554873 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.554689    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdnfm\" (UniqueName: \"kubernetes.io/projected/89b4886d-bfd4-4468-9db7-cd87445e24aa-kube-api-access-fdnfm\") pod \"e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") " pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:46.555031 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.554956    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-home\") pod \"e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") " pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:46.555031 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.554960    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-tmp-dir\") pod \"e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") " pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:46.555031 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.555021    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-model-cache\") pod \"e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") " pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:46.556959 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.556928    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-dshm\") pod \"e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") " pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:46.557109 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.557092    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89b4886d-bfd4-4468-9db7-cd87445e24aa-tls-certs\") pod \"e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") " pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:46.563656 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.563634    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdnfm\" (UniqueName: \"kubernetes.io/projected/89b4886d-bfd4-4468-9db7-cd87445e24aa-kube-api-access-fdnfm\") pod \"e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") " pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:46.620188 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.620148    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:46.750244 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:46.750198    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"]
Jul 28 12:09:46.753965 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:09:46.753935    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89b4886d_bfd4_4468_9db7_cd87445e24aa.slice/crio-864aa014de6a08a3ac2dabc5c16af7c7548dd1eead195e1fc89de52c9da8261a WatchSource:0}: Error finding container 864aa014de6a08a3ac2dabc5c16af7c7548dd1eead195e1fc89de52c9da8261a: Status 404 returned error can't find the container with id 864aa014de6a08a3ac2dabc5c16af7c7548dd1eead195e1fc89de52c9da8261a
Jul 28 12:09:47.639595 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:47.639558    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw" event={"ID":"89b4886d-bfd4-4468-9db7-cd87445e24aa","Type":"ContainerStarted","Data":"20939287af4bc2283a308ff12d010bd367ddca095c0e87474eff41d65dd84e73"}
Jul 28 12:09:47.639976 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:47.639606    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw" event={"ID":"89b4886d-bfd4-4468-9db7-cd87445e24aa","Type":"ContainerStarted","Data":"864aa014de6a08a3ac2dabc5c16af7c7548dd1eead195e1fc89de52c9da8261a"}
Jul 28 12:09:47.671744 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:47.671695    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw" podStartSLOduration=1.671679046 podStartE2EDuration="1.671679046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:09:47.671243297 +0000 UTC m=+2339.529899324" watchObservedRunningTime="2026-07-28 12:09:47.671679046 +0000 UTC m=+2339.530335044"
Jul 28 12:09:48.601137 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:48.601108    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx"
Jul 28 12:09:49.605455 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:49.605427    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v"
Jul 28 12:09:49.671167 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:49.671135    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:09:56.620850 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:56.620808    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:56.621254 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:56.620872    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:56.633160 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:56.633137    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:09:56.691405 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:09:56.691381    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:10:15.132286 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:15.132245    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:10:34.931478 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:34.931444    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:10:36.274950 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.274918    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"]
Jul 28 12:10:36.275446 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.275131    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn" podUID="f51d7381-2d43-46c0-b798-96e8c9425869" containerName="maas-api" containerID="cri-o://f289b83cb8af6cf80c974bea7fff25d4ead6a05482355bcec6aa0d601c0544d8" gracePeriod=30
Jul 28 12:10:36.333103 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.333070    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v"]
Jul 28 12:10:36.333390 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.333327    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v" podUID="d2db8331-1573-4770-9798-437f930f131d" containerName="payload-processing" containerID="cri-o://51dd7a2a10697265d5afbe3daace95e74596fad13a68de02d4639367adb55b66" gracePeriod=30
Jul 28 12:10:36.340160 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.340074    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx"]
Jul 28 12:10:36.340385 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.340297    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx" podUID="dcd9b4f6-6b04-4b3f-af07-628352d67ca6" containerName="payload-pre-processing" containerID="cri-o://298a243e50aee30294cffd0bba75fe785799f1e531522739a103873e595f208f" gracePeriod=30
Jul 28 12:10:36.584522 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.584492    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"
Jul 28 12:10:36.617147 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.617122    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v"
Jul 28 12:10:36.625302 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.625280    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx"
Jul 28 12:10:36.681759 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.681725    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knjqb\" (UniqueName: \"kubernetes.io/projected/d2db8331-1573-4770-9798-437f930f131d-kube-api-access-knjqb\") pod \"d2db8331-1573-4770-9798-437f930f131d\" (UID: \"d2db8331-1573-4770-9798-437f930f131d\") "
Jul 28 12:10:36.681921 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.681776    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrcdj\" (UniqueName: \"kubernetes.io/projected/f51d7381-2d43-46c0-b798-96e8c9425869-kube-api-access-mrcdj\") pod \"f51d7381-2d43-46c0-b798-96e8c9425869\" (UID: \"f51d7381-2d43-46c0-b798-96e8c9425869\") "
Jul 28 12:10:36.681921 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.681906    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f51d7381-2d43-46c0-b798-96e8c9425869-maas-api-tls\") pod \"f51d7381-2d43-46c0-b798-96e8c9425869\" (UID: \"f51d7381-2d43-46c0-b798-96e8c9425869\") "
Jul 28 12:10:36.682047 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.681961    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51d7381-2d43-46c0-b798-96e8c9425869-openshift-service-ca\") pod \"f51d7381-2d43-46c0-b798-96e8c9425869\" (UID: \"f51d7381-2d43-46c0-b798-96e8c9425869\") "
Jul 28 12:10:36.682047 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.682000    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2db8331-1573-4770-9798-437f930f131d-plugins-config-volume\") pod \"d2db8331-1573-4770-9798-437f930f131d\" (UID: \"d2db8331-1573-4770-9798-437f930f131d\") "
Jul 28 12:10:36.682548 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.682388    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51d7381-2d43-46c0-b798-96e8c9425869-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f51d7381-2d43-46c0-b798-96e8c9425869" (UID: "f51d7381-2d43-46c0-b798-96e8c9425869"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:10:36.682666 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.682602    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2db8331-1573-4770-9798-437f930f131d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d2db8331-1573-4770-9798-437f930f131d" (UID: "d2db8331-1573-4770-9798-437f930f131d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:10:36.684076 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.684056    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2db8331-1573-4770-9798-437f930f131d-kube-api-access-knjqb" (OuterVolumeSpecName: "kube-api-access-knjqb") pod "d2db8331-1573-4770-9798-437f930f131d" (UID: "d2db8331-1573-4770-9798-437f930f131d"). InnerVolumeSpecName "kube-api-access-knjqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:10:36.684341 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.684317    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f51d7381-2d43-46c0-b798-96e8c9425869-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f51d7381-2d43-46c0-b798-96e8c9425869" (UID: "f51d7381-2d43-46c0-b798-96e8c9425869"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:10:36.684341 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.684331    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51d7381-2d43-46c0-b798-96e8c9425869-kube-api-access-mrcdj" (OuterVolumeSpecName: "kube-api-access-mrcdj") pod "f51d7381-2d43-46c0-b798-96e8c9425869" (UID: "f51d7381-2d43-46c0-b798-96e8c9425869"). InnerVolumeSpecName "kube-api-access-mrcdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:10:36.782826 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.782798    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8rzq\" (UniqueName: \"kubernetes.io/projected/dcd9b4f6-6b04-4b3f-af07-628352d67ca6-kube-api-access-x8rzq\") pod \"dcd9b4f6-6b04-4b3f-af07-628352d67ca6\" (UID: \"dcd9b4f6-6b04-4b3f-af07-628352d67ca6\") "
Jul 28 12:10:36.782994 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.782835    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcd9b4f6-6b04-4b3f-af07-628352d67ca6-plugins-config-volume\") pod \"dcd9b4f6-6b04-4b3f-af07-628352d67ca6\" (UID: \"dcd9b4f6-6b04-4b3f-af07-628352d67ca6\") "
Jul 28 12:10:36.782994 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.782987    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrcdj\" (UniqueName: \"kubernetes.io/projected/f51d7381-2d43-46c0-b798-96e8c9425869-kube-api-access-mrcdj\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:10:36.783116 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.782998    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f51d7381-2d43-46c0-b798-96e8c9425869-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:10:36.783116 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.783008    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51d7381-2d43-46c0-b798-96e8c9425869-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:10:36.783116 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.783017    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2db8331-1573-4770-9798-437f930f131d-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:10:36.783116 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.783025    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knjqb\" (UniqueName: \"kubernetes.io/projected/d2db8331-1573-4770-9798-437f930f131d-kube-api-access-knjqb\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:10:36.783326 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.783273    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd9b4f6-6b04-4b3f-af07-628352d67ca6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dcd9b4f6-6b04-4b3f-af07-628352d67ca6" (UID: "dcd9b4f6-6b04-4b3f-af07-628352d67ca6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:10:36.784779 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.784757    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd9b4f6-6b04-4b3f-af07-628352d67ca6-kube-api-access-x8rzq" (OuterVolumeSpecName: "kube-api-access-x8rzq") pod "dcd9b4f6-6b04-4b3f-af07-628352d67ca6" (UID: "dcd9b4f6-6b04-4b3f-af07-628352d67ca6"). InnerVolumeSpecName "kube-api-access-x8rzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:10:36.857418 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.857334    2561 generic.go:356] "Generic (PLEG): container finished" podID="f51d7381-2d43-46c0-b798-96e8c9425869" containerID="f289b83cb8af6cf80c974bea7fff25d4ead6a05482355bcec6aa0d601c0544d8" exitCode=0
Jul 28 12:10:36.857418 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.857396    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"
Jul 28 12:10:36.857682 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.857430    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn" event={"ID":"f51d7381-2d43-46c0-b798-96e8c9425869","Type":"ContainerDied","Data":"f289b83cb8af6cf80c974bea7fff25d4ead6a05482355bcec6aa0d601c0544d8"}
Jul 28 12:10:36.857682 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.857524    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn" event={"ID":"f51d7381-2d43-46c0-b798-96e8c9425869","Type":"ContainerDied","Data":"4e07ed03f939c82982bab61e28ac425ff7e01f2c7f7e41b7a553b7f479606289"}
Jul 28 12:10:36.857682 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.857555    2561 scope.go:117] "RemoveContainer" containerID="f289b83cb8af6cf80c974bea7fff25d4ead6a05482355bcec6aa0d601c0544d8"
Jul 28 12:10:36.858776 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.858728    2561 generic.go:356] "Generic (PLEG): container finished" podID="d2db8331-1573-4770-9798-437f930f131d" containerID="51dd7a2a10697265d5afbe3daace95e74596fad13a68de02d4639367adb55b66" exitCode=0
Jul 28 12:10:36.858776 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.858768    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v"
Jul 28 12:10:36.858923 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.858808    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v" event={"ID":"d2db8331-1573-4770-9798-437f930f131d","Type":"ContainerDied","Data":"51dd7a2a10697265d5afbe3daace95e74596fad13a68de02d4639367adb55b66"}
Jul 28 12:10:36.858923 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.858859    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v" event={"ID":"d2db8331-1573-4770-9798-437f930f131d","Type":"ContainerDied","Data":"2d1d872c7781db91af93e1195f911a4b844eb80b4b620daa67d6a60d32e88361"}
Jul 28 12:10:36.860229 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.860205    2561 generic.go:356] "Generic (PLEG): container finished" podID="dcd9b4f6-6b04-4b3f-af07-628352d67ca6" containerID="298a243e50aee30294cffd0bba75fe785799f1e531522739a103873e595f208f" exitCode=0
Jul 28 12:10:36.860369 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.860250    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx"
Jul 28 12:10:36.860369 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.860272    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx" event={"ID":"dcd9b4f6-6b04-4b3f-af07-628352d67ca6","Type":"ContainerDied","Data":"298a243e50aee30294cffd0bba75fe785799f1e531522739a103873e595f208f"}
Jul 28 12:10:36.860369 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.860307    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx" event={"ID":"dcd9b4f6-6b04-4b3f-af07-628352d67ca6","Type":"ContainerDied","Data":"96103ae22e3dd99279b3bf1ebc6ae5d7dfd98dd897e31909f577dd5c950eb840"}
Jul 28 12:10:36.866515 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.866499    2561 scope.go:117] "RemoveContainer" containerID="f289b83cb8af6cf80c974bea7fff25d4ead6a05482355bcec6aa0d601c0544d8"
Jul 28 12:10:36.866805 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:10:36.866771    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f289b83cb8af6cf80c974bea7fff25d4ead6a05482355bcec6aa0d601c0544d8\": container with ID starting with f289b83cb8af6cf80c974bea7fff25d4ead6a05482355bcec6aa0d601c0544d8 not found: ID does not exist" containerID="f289b83cb8af6cf80c974bea7fff25d4ead6a05482355bcec6aa0d601c0544d8"
Jul 28 12:10:36.867400 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.866808    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f289b83cb8af6cf80c974bea7fff25d4ead6a05482355bcec6aa0d601c0544d8"} err="failed to get container status \"f289b83cb8af6cf80c974bea7fff25d4ead6a05482355bcec6aa0d601c0544d8\": rpc error: code = NotFound desc = could not find container \"f289b83cb8af6cf80c974bea7fff25d4ead6a05482355bcec6aa0d601c0544d8\": container with ID starting with f289b83cb8af6cf80c974bea7fff25d4ead6a05482355bcec6aa0d601c0544d8 not found: ID does not exist"
Jul 28 12:10:36.867400 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.866830    2561 scope.go:117] "RemoveContainer" containerID="51dd7a2a10697265d5afbe3daace95e74596fad13a68de02d4639367adb55b66"
Jul 28 12:10:36.877312 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.877293    2561 scope.go:117] "RemoveContainer" containerID="51dd7a2a10697265d5afbe3daace95e74596fad13a68de02d4639367adb55b66"
Jul 28 12:10:36.877559 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:10:36.877542    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51dd7a2a10697265d5afbe3daace95e74596fad13a68de02d4639367adb55b66\": container with ID starting with 51dd7a2a10697265d5afbe3daace95e74596fad13a68de02d4639367adb55b66 not found: ID does not exist" containerID="51dd7a2a10697265d5afbe3daace95e74596fad13a68de02d4639367adb55b66"
Jul 28 12:10:36.877653 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.877564    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51dd7a2a10697265d5afbe3daace95e74596fad13a68de02d4639367adb55b66"} err="failed to get container status \"51dd7a2a10697265d5afbe3daace95e74596fad13a68de02d4639367adb55b66\": rpc error: code = NotFound desc = could not find container \"51dd7a2a10697265d5afbe3daace95e74596fad13a68de02d4639367adb55b66\": container with ID starting with 51dd7a2a10697265d5afbe3daace95e74596fad13a68de02d4639367adb55b66 not found: ID does not exist"
Jul 28 12:10:36.877653 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.877596    2561 scope.go:117] "RemoveContainer" containerID="298a243e50aee30294cffd0bba75fe785799f1e531522739a103873e595f208f"
Jul 28 12:10:36.882434 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.882415    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"]
Jul 28 12:10:36.884033 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.884012    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8rzq\" (UniqueName: \"kubernetes.io/projected/dcd9b4f6-6b04-4b3f-af07-628352d67ca6-kube-api-access-x8rzq\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:10:36.884033 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.884033    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcd9b4f6-6b04-4b3f-af07-628352d67ca6-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:10:36.884808 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.884708    2561 scope.go:117] "RemoveContainer" containerID="298a243e50aee30294cffd0bba75fe785799f1e531522739a103873e595f208f"
Jul 28 12:10:36.885144 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:10:36.885120    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"298a243e50aee30294cffd0bba75fe785799f1e531522739a103873e595f208f\": container with ID starting with 298a243e50aee30294cffd0bba75fe785799f1e531522739a103873e595f208f not found: ID does not exist" containerID="298a243e50aee30294cffd0bba75fe785799f1e531522739a103873e595f208f"
Jul 28 12:10:36.885308 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.885270    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298a243e50aee30294cffd0bba75fe785799f1e531522739a103873e595f208f"} err="failed to get container status \"298a243e50aee30294cffd0bba75fe785799f1e531522739a103873e595f208f\": rpc error: code = NotFound desc = could not find container \"298a243e50aee30294cffd0bba75fe785799f1e531522739a103873e595f208f\": container with ID starting with 298a243e50aee30294cffd0bba75fe785799f1e531522739a103873e595f208f not found: ID does not exist"
Jul 28 12:10:36.886530 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.886513    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7038fa7e-cf9b9d7f4-qw9vn"]
Jul 28 12:10:36.896160 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.896139    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx"]
Jul 28 12:10:36.902855 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.902835    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7038fa7e-864b77fd5b-46fjx"]
Jul 28 12:10:36.914623 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.914604    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v"]
Jul 28 12:10:36.917739 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:36.917721    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7038fa7e-7fc75b959d-mq22v"]
Jul 28 12:10:38.545725 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:38.545697    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2db8331-1573-4770-9798-437f930f131d" path="/var/lib/kubelet/pods/d2db8331-1573-4770-9798-437f930f131d/volumes"
Jul 28 12:10:38.546098 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:38.546035    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcd9b4f6-6b04-4b3f-af07-628352d67ca6" path="/var/lib/kubelet/pods/dcd9b4f6-6b04-4b3f-af07-628352d67ca6/volumes"
Jul 28 12:10:38.546362 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:38.546350    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f51d7381-2d43-46c0-b798-96e8c9425869" path="/var/lib/kubelet/pods/f51d7381-2d43-46c0-b798-96e8c9425869/volumes"
Jul 28 12:10:44.935909 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:44.935876    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:10:48.607359 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:48.607329    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"]
Jul 28 12:10:48.707916 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:10:48.707891    2561 scope.go:117] "RemoveContainer" containerID="fd04e82b5b738e580b135a234166d918f5d74f4bca44eba099b02e559f55fc74"
Jul 28 12:11:03.629825 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.629781    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw" podUID="89b4886d-bfd4-4468-9db7-cd87445e24aa" containerName="main" containerID="cri-o://20939287af4bc2283a308ff12d010bd367ddca095c0e87474eff41d65dd84e73" gracePeriod=45
Jul 28 12:11:03.865369 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.865342    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:11:03.897547 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.897491    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-model-cache\") pod \"89b4886d-bfd4-4468-9db7-cd87445e24aa\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") "
Jul 28 12:11:03.897547 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.897538    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-dshm\") pod \"89b4886d-bfd4-4468-9db7-cd87445e24aa\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") "
Jul 28 12:11:03.897762 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.897604    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-tmp-dir\") pod \"89b4886d-bfd4-4468-9db7-cd87445e24aa\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") "
Jul 28 12:11:03.897762 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.897641    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-home\") pod \"89b4886d-bfd4-4468-9db7-cd87445e24aa\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") "
Jul 28 12:11:03.897762 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.897681    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdnfm\" (UniqueName: \"kubernetes.io/projected/89b4886d-bfd4-4468-9db7-cd87445e24aa-kube-api-access-fdnfm\") pod \"89b4886d-bfd4-4468-9db7-cd87445e24aa\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") "
Jul 28 12:11:03.897762 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.897710    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89b4886d-bfd4-4468-9db7-cd87445e24aa-tls-certs\") pod \"89b4886d-bfd4-4468-9db7-cd87445e24aa\" (UID: \"89b4886d-bfd4-4468-9db7-cd87445e24aa\") "
Jul 28 12:11:03.897978 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.897809    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "89b4886d-bfd4-4468-9db7-cd87445e24aa" (UID: "89b4886d-bfd4-4468-9db7-cd87445e24aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:11:03.897978 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.897859    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89b4886d-bfd4-4468-9db7-cd87445e24aa" (UID: "89b4886d-bfd4-4468-9db7-cd87445e24aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:11:03.897978 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.897901    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-home" (OuterVolumeSpecName: "home") pod "89b4886d-bfd4-4468-9db7-cd87445e24aa" (UID: "89b4886d-bfd4-4468-9db7-cd87445e24aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:11:03.898157 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.897992    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-model-cache\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:11:03.898157 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.898009    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-tmp-dir\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:11:03.898157 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.898017    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-home\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:11:03.899905 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.899875    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-dshm" (OuterVolumeSpecName: "dshm") pod "89b4886d-bfd4-4468-9db7-cd87445e24aa" (UID: "89b4886d-bfd4-4468-9db7-cd87445e24aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:11:03.900006 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.899902    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b4886d-bfd4-4468-9db7-cd87445e24aa-kube-api-access-fdnfm" (OuterVolumeSpecName: "kube-api-access-fdnfm") pod "89b4886d-bfd4-4468-9db7-cd87445e24aa" (UID: "89b4886d-bfd4-4468-9db7-cd87445e24aa"). InnerVolumeSpecName "kube-api-access-fdnfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:11:03.900006 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.899911    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b4886d-bfd4-4468-9db7-cd87445e24aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89b4886d-bfd4-4468-9db7-cd87445e24aa" (UID: "89b4886d-bfd4-4468-9db7-cd87445e24aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:11:03.984922 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.984889    2561 generic.go:356] "Generic (PLEG): container finished" podID="89b4886d-bfd4-4468-9db7-cd87445e24aa" containerID="20939287af4bc2283a308ff12d010bd367ddca095c0e87474eff41d65dd84e73" exitCode=0
Jul 28 12:11:03.985055 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.984936    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"
Jul 28 12:11:03.985055 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.984944    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw" event={"ID":"89b4886d-bfd4-4468-9db7-cd87445e24aa","Type":"ContainerDied","Data":"20939287af4bc2283a308ff12d010bd367ddca095c0e87474eff41d65dd84e73"}
Jul 28 12:11:03.985055 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.984985    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw" event={"ID":"89b4886d-bfd4-4468-9db7-cd87445e24aa","Type":"ContainerDied","Data":"864aa014de6a08a3ac2dabc5c16af7c7548dd1eead195e1fc89de52c9da8261a"}
Jul 28 12:11:03.985055 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.985006    2561 scope.go:117] "RemoveContainer" containerID="20939287af4bc2283a308ff12d010bd367ddca095c0e87474eff41d65dd84e73"
Jul 28 12:11:03.993737 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.993630    2561 scope.go:117] "RemoveContainer" containerID="20939287af4bc2283a308ff12d010bd367ddca095c0e87474eff41d65dd84e73"
Jul 28 12:11:03.993940 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:11:03.993912    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20939287af4bc2283a308ff12d010bd367ddca095c0e87474eff41d65dd84e73\": container with ID starting with 20939287af4bc2283a308ff12d010bd367ddca095c0e87474eff41d65dd84e73 not found: ID does not exist" containerID="20939287af4bc2283a308ff12d010bd367ddca095c0e87474eff41d65dd84e73"
Jul 28 12:11:03.994059 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.993949    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20939287af4bc2283a308ff12d010bd367ddca095c0e87474eff41d65dd84e73"} err="failed to get container status \"20939287af4bc2283a308ff12d010bd367ddca095c0e87474eff41d65dd84e73\": rpc error: code = NotFound desc = could not find container \"20939287af4bc2283a308ff12d010bd367ddca095c0e87474eff41d65dd84e73\": container with ID starting with 20939287af4bc2283a308ff12d010bd367ddca095c0e87474eff41d65dd84e73 not found: ID does not exist"
Jul 28 12:11:03.998537 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.998516    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdnfm\" (UniqueName: \"kubernetes.io/projected/89b4886d-bfd4-4468-9db7-cd87445e24aa-kube-api-access-fdnfm\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:11:03.998655 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.998540    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89b4886d-bfd4-4468-9db7-cd87445e24aa-tls-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:11:03.998655 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:03.998554    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89b4886d-bfd4-4468-9db7-cd87445e24aa-dshm\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:11:04.008080 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:04.008060    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"]
Jul 28 12:11:04.013454 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:04.013435    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7038fa7e/e2e-lifecycle-model-7038fa7e-kserve-5cd6787f9-h7bcw"]
Jul 28 12:11:04.546296 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:04.546253    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89b4886d-bfd4-4468-9db7-cd87445e24aa" path="/var/lib/kubelet/pods/89b4886d-bfd4-4468-9db7-cd87445e24aa/volumes"
Jul 28 12:11:36.534324 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:36.534290    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:11:46.538134 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:11:46.538089    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:12:33.282500 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.282464    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"]
Jul 28 12:12:33.283006 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.282820    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2db8331-1573-4770-9798-437f930f131d" containerName="payload-processing"
Jul 28 12:12:33.283006 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.282833    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2db8331-1573-4770-9798-437f930f131d" containerName="payload-processing"
Jul 28 12:12:33.283006 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.282842    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89b4886d-bfd4-4468-9db7-cd87445e24aa" containerName="main"
Jul 28 12:12:33.283006 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.282848    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="89b4886d-bfd4-4468-9db7-cd87445e24aa" containerName="main"
Jul 28 12:12:33.283006 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.282853    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd9b4f6-6b04-4b3f-af07-628352d67ca6" containerName="payload-pre-processing"
Jul 28 12:12:33.283006 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.282859    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd9b4f6-6b04-4b3f-af07-628352d67ca6" containerName="payload-pre-processing"
Jul 28 12:12:33.283006 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.282874    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f51d7381-2d43-46c0-b798-96e8c9425869" containerName="maas-api"
Jul 28 12:12:33.283006 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.282879    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f51d7381-2d43-46c0-b798-96e8c9425869" containerName="maas-api"
Jul 28 12:12:33.283006 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.282926    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcd9b4f6-6b04-4b3f-af07-628352d67ca6" containerName="payload-pre-processing"
Jul 28 12:12:33.283006 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.282935    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f51d7381-2d43-46c0-b798-96e8c9425869" containerName="maas-api"
Jul 28 12:12:33.283006 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.282941    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2db8331-1573-4770-9798-437f930f131d" containerName="payload-processing"
Jul 28 12:12:33.283006 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.282948    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="89b4886d-bfd4-4468-9db7-cd87445e24aa" containerName="main"
Jul 28 12:12:33.285831 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.285815    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"
Jul 28 12:12:33.288810 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.288791    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-21898b-serving-cert\""
Jul 28 12:12:33.288810 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.288799    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xchcw\""
Jul 28 12:12:33.289445 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.289425    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 12:12:33.289518 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.289470    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 12:12:33.296980 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.296959    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"]
Jul 28 12:12:33.419342 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.419316    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33087b5b-9f07-46da-9564-9f79dcbf4813-maas-api-tls\") pod \"maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7\" (UID: \"33087b5b-9f07-46da-9564-9f79dcbf4813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"
Jul 28 12:12:33.419458 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.419344    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33087b5b-9f07-46da-9564-9f79dcbf4813-openshift-service-ca\") pod \"maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7\" (UID: \"33087b5b-9f07-46da-9564-9f79dcbf4813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"
Jul 28 12:12:33.419458 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.419363    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brcj5\" (UniqueName: \"kubernetes.io/projected/33087b5b-9f07-46da-9564-9f79dcbf4813-kube-api-access-brcj5\") pod \"maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7\" (UID: \"33087b5b-9f07-46da-9564-9f79dcbf4813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"
Jul 28 12:12:33.520278 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.520248    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33087b5b-9f07-46da-9564-9f79dcbf4813-maas-api-tls\") pod \"maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7\" (UID: \"33087b5b-9f07-46da-9564-9f79dcbf4813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"
Jul 28 12:12:33.520391 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.520281    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33087b5b-9f07-46da-9564-9f79dcbf4813-openshift-service-ca\") pod \"maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7\" (UID: \"33087b5b-9f07-46da-9564-9f79dcbf4813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"
Jul 28 12:12:33.520391 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.520306    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brcj5\" (UniqueName: \"kubernetes.io/projected/33087b5b-9f07-46da-9564-9f79dcbf4813-kube-api-access-brcj5\") pod \"maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7\" (UID: \"33087b5b-9f07-46da-9564-9f79dcbf4813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"
Jul 28 12:12:33.520478 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:12:33.520387    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-serving-cert: secret "maas-api-e2e-inf-a-21898b-serving-cert" not found
Jul 28 12:12:33.520478 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:12:33.520455    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33087b5b-9f07-46da-9564-9f79dcbf4813-maas-api-tls podName:33087b5b-9f07-46da-9564-9f79dcbf4813 nodeName:}" failed. No retries permitted until 2026-07-28 12:12:34.02043764 +0000 UTC m=+2505.879093622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33087b5b-9f07-46da-9564-9f79dcbf4813-maas-api-tls") pod "maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7" (UID: "33087b5b-9f07-46da-9564-9f79dcbf4813") : secret "maas-api-e2e-inf-a-21898b-serving-cert" not found
Jul 28 12:12:33.520862 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.520845    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33087b5b-9f07-46da-9564-9f79dcbf4813-openshift-service-ca\") pod \"maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7\" (UID: \"33087b5b-9f07-46da-9564-9f79dcbf4813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"
Jul 28 12:12:33.528715 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:33.528697    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brcj5\" (UniqueName: \"kubernetes.io/projected/33087b5b-9f07-46da-9564-9f79dcbf4813-kube-api-access-brcj5\") pod \"maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7\" (UID: \"33087b5b-9f07-46da-9564-9f79dcbf4813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"
Jul 28 12:12:34.024203 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.024172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33087b5b-9f07-46da-9564-9f79dcbf4813-maas-api-tls\") pod \"maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7\" (UID: \"33087b5b-9f07-46da-9564-9f79dcbf4813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"
Jul 28 12:12:34.026475 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.026449    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33087b5b-9f07-46da-9564-9f79dcbf4813-maas-api-tls\") pod \"maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7\" (UID: \"33087b5b-9f07-46da-9564-9f79dcbf4813\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"
Jul 28 12:12:34.121822 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.121787    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq"]
Jul 28 12:12:34.125071 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.125057    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq"
Jul 28 12:12:34.127554 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.127535    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-21898b\""
Jul 28 12:12:34.127757 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.127731    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 12:12:34.127757 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.127750    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-21898b-dockercfg-xpdk4\""
Jul 28 12:12:34.127896 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.127838    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 12:12:34.134090 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.134071    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq"]
Jul 28 12:12:34.195257 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.195230    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"
Jul 28 12:12:34.225902 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.225876    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf7l2\" (UniqueName: \"kubernetes.io/projected/e96f10b6-2b18-404f-bb89-6ca66fa49533-kube-api-access-xf7l2\") pod \"payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq\" (UID: \"e96f10b6-2b18-404f-bb89-6ca66fa49533\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq"
Jul 28 12:12:34.226028 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.225947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96f10b6-2b18-404f-bb89-6ca66fa49533-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq\" (UID: \"e96f10b6-2b18-404f-bb89-6ca66fa49533\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq"
Jul 28 12:12:34.243041 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.243014    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn"]
Jul 28 12:12:34.246848 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.246825    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn"
Jul 28 12:12:34.254055 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.254024    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn"]
Jul 28 12:12:34.326898 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.326869    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96f10b6-2b18-404f-bb89-6ca66fa49533-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq\" (UID: \"e96f10b6-2b18-404f-bb89-6ca66fa49533\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq"
Jul 28 12:12:34.327237 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.326914    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936cd400-18a9-421f-900f-d026de89a7bc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn\" (UID: \"936cd400-18a9-421f-900f-d026de89a7bc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn"
Jul 28 12:12:34.327237 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.326962    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf7l2\" (UniqueName: \"kubernetes.io/projected/e96f10b6-2b18-404f-bb89-6ca66fa49533-kube-api-access-xf7l2\") pod \"payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq\" (UID: \"e96f10b6-2b18-404f-bb89-6ca66fa49533\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq"
Jul 28 12:12:34.327237 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.326978    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhfjp\" (UniqueName: \"kubernetes.io/projected/936cd400-18a9-421f-900f-d026de89a7bc-kube-api-access-zhfjp\") pod \"payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn\" (UID: \"936cd400-18a9-421f-900f-d026de89a7bc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn"
Jul 28 12:12:34.327524 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.327506    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96f10b6-2b18-404f-bb89-6ca66fa49533-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq\" (UID: \"e96f10b6-2b18-404f-bb89-6ca66fa49533\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq"
Jul 28 12:12:34.330380 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.330361    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"]
Jul 28 12:12:34.332424 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:12:34.332391    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33087b5b_9f07_46da_9564_9f79dcbf4813.slice/crio-99140b0230748bbaae2b0f1d965ec08867292cdd5e55d6d05d91fddd09d3a28b WatchSource:0}: Error finding container 99140b0230748bbaae2b0f1d965ec08867292cdd5e55d6d05d91fddd09d3a28b: Status 404 returned error can't find the container with id 99140b0230748bbaae2b0f1d965ec08867292cdd5e55d6d05d91fddd09d3a28b
Jul 28 12:12:34.334073 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.334051    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:12:34.334993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.334976    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf7l2\" (UniqueName: \"kubernetes.io/projected/e96f10b6-2b18-404f-bb89-6ca66fa49533-kube-api-access-xf7l2\") pod \"payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq\" (UID: \"e96f10b6-2b18-404f-bb89-6ca66fa49533\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq"
Jul 28 12:12:34.359558 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.359532    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7" event={"ID":"33087b5b-9f07-46da-9564-9f79dcbf4813","Type":"ContainerStarted","Data":"99140b0230748bbaae2b0f1d965ec08867292cdd5e55d6d05d91fddd09d3a28b"}
Jul 28 12:12:34.428026 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.428001    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhfjp\" (UniqueName: \"kubernetes.io/projected/936cd400-18a9-421f-900f-d026de89a7bc-kube-api-access-zhfjp\") pod \"payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn\" (UID: \"936cd400-18a9-421f-900f-d026de89a7bc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn"
Jul 28 12:12:34.428123 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.428056    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936cd400-18a9-421f-900f-d026de89a7bc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn\" (UID: \"936cd400-18a9-421f-900f-d026de89a7bc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn"
Jul 28 12:12:34.428627 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.428612    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936cd400-18a9-421f-900f-d026de89a7bc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn\" (UID: \"936cd400-18a9-421f-900f-d026de89a7bc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn"
Jul 28 12:12:34.435159 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.435139    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq"
Jul 28 12:12:34.436877 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.436860    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhfjp\" (UniqueName: \"kubernetes.io/projected/936cd400-18a9-421f-900f-d026de89a7bc-kube-api-access-zhfjp\") pod \"payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn\" (UID: \"936cd400-18a9-421f-900f-d026de89a7bc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn"
Jul 28 12:12:34.558498 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.558475    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq"]
Jul 28 12:12:34.560840 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:12:34.560812    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96f10b6_2b18_404f_bb89_6ca66fa49533.slice/crio-4a8c68b199de34f890f552060748f7318eb89a7b289cadd558491a699e90a36c WatchSource:0}: Error finding container 4a8c68b199de34f890f552060748f7318eb89a7b289cadd558491a699e90a36c: Status 404 returned error can't find the container with id 4a8c68b199de34f890f552060748f7318eb89a7b289cadd558491a699e90a36c
Jul 28 12:12:34.563641 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.563623    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn"
Jul 28 12:12:34.697058 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:34.697036    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn"]
Jul 28 12:12:34.699282 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:12:34.699252    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936cd400_18a9_421f_900f_d026de89a7bc.slice/crio-71492131c19b7ee290a7f549719d35a1ec47f8367258680ae9a21f7d147cc3a1 WatchSource:0}: Error finding container 71492131c19b7ee290a7f549719d35a1ec47f8367258680ae9a21f7d147cc3a1: Status 404 returned error can't find the container with id 71492131c19b7ee290a7f549719d35a1ec47f8367258680ae9a21f7d147cc3a1
Jul 28 12:12:35.364967 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:35.364928    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7" event={"ID":"33087b5b-9f07-46da-9564-9f79dcbf4813","Type":"ContainerStarted","Data":"c1b5a0b1f8065bd30753e0fdbd5608ee90f29ec86a043464531f7fc57c9e8193"}
Jul 28 12:12:35.365415 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:35.365090    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"
Jul 28 12:12:35.366624 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:35.366599    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn" event={"ID":"936cd400-18a9-421f-900f-d026de89a7bc","Type":"ContainerStarted","Data":"06fc64694a7bcfde6c23e7de7abb52bba14b15f03d4ac180fe30424fb2cf4a66"}
Jul 28 12:12:35.366767 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:35.366647    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn" event={"ID":"936cd400-18a9-421f-900f-d026de89a7bc","Type":"ContainerStarted","Data":"71492131c19b7ee290a7f549719d35a1ec47f8367258680ae9a21f7d147cc3a1"}
Jul 28 12:12:35.366767 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:35.366674    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn"
Jul 28 12:12:35.368268 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:35.368235    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq" event={"ID":"e96f10b6-2b18-404f-bb89-6ca66fa49533","Type":"ContainerStarted","Data":"b427d052653f929db79a6f66b5d10af9ab64499e1c518760e4625b9d08dd9b30"}
Jul 28 12:12:35.368377 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:35.368278    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq" event={"ID":"e96f10b6-2b18-404f-bb89-6ca66fa49533","Type":"ContainerStarted","Data":"4a8c68b199de34f890f552060748f7318eb89a7b289cadd558491a699e90a36c"}
Jul 28 12:12:35.368377 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:35.368316    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq"
Jul 28 12:12:35.382401 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:35.382344    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7" podStartSLOduration=1.807511503 podStartE2EDuration="2.382328642s" totalImagesPullingTime="574.817139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:12:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:12:35.380616495 +0000 UTC m=+2507.239272495" watchObservedRunningTime="2026-07-28 12:12:35.382328642 +0000 UTC m=+2507.240984648"
Jul 28 12:12:35.396374 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:35.396330    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn" podStartSLOduration=1.396315715 podStartE2EDuration="1.396315715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:12:35.395912477 +0000 UTC m=+2507.254568476" watchObservedRunningTime="2026-07-28 12:12:35.396315715 +0000 UTC m=+2507.254971717"
Jul 28 12:12:35.410192 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:35.410137    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq" podStartSLOduration=1.410126051 podStartE2EDuration="1.410126051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:12:35.409746949 +0000 UTC m=+2507.268402950" watchObservedRunningTime="2026-07-28 12:12:35.410126051 +0000 UTC m=+2507.268782051"
Jul 28 12:12:37.730879 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:37.730843    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:12:41.343093 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:41.343011    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"]
Jul 28 12:12:41.347195 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:41.347173    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"
Jul 28 12:12:41.349400 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:41.349380    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-44546f-serving-cert\""
Jul 28 12:12:41.360359 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:41.360334    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"]
Jul 28 12:12:41.378386 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:41.378361    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"
Jul 28 12:12:41.485708 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:41.485676    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b172ed58-35b3-4db8-b00f-17c30e6befcd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk\" (UID: \"b172ed58-35b3-4db8-b00f-17c30e6befcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"
Jul 28 12:12:41.485900 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:41.485713    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b172ed58-35b3-4db8-b00f-17c30e6befcd-maas-api-tls\") pod \"maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk\" (UID: \"b172ed58-35b3-4db8-b00f-17c30e6befcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"
Jul 28 12:12:41.485900 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:41.485782    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfw2s\" (UniqueName: \"kubernetes.io/projected/b172ed58-35b3-4db8-b00f-17c30e6befcd-kube-api-access-wfw2s\") pod \"maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk\" (UID: \"b172ed58-35b3-4db8-b00f-17c30e6befcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"
Jul 28 12:12:41.586930 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:41.586898    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b172ed58-35b3-4db8-b00f-17c30e6befcd-maas-api-tls\") pod \"maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk\" (UID: \"b172ed58-35b3-4db8-b00f-17c30e6befcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"
Jul 28 12:12:41.587101 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:41.586946    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfw2s\" (UniqueName: \"kubernetes.io/projected/b172ed58-35b3-4db8-b00f-17c30e6befcd-kube-api-access-wfw2s\") pod \"maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk\" (UID: \"b172ed58-35b3-4db8-b00f-17c30e6befcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"
Jul 28 12:12:41.587101 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:41.587018    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b172ed58-35b3-4db8-b00f-17c30e6befcd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk\" (UID: \"b172ed58-35b3-4db8-b00f-17c30e6befcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"
Jul 28 12:12:41.587101 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:12:41.587057    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-serving-cert: secret "maas-api-e2e-inf-b-44546f-serving-cert" not found
Jul 28 12:12:41.587203 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:12:41.587126    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b172ed58-35b3-4db8-b00f-17c30e6befcd-maas-api-tls podName:b172ed58-35b3-4db8-b00f-17c30e6befcd nodeName:}" failed. No retries permitted until 2026-07-28 12:12:42.087108924 +0000 UTC m=+2513.945764904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b172ed58-35b3-4db8-b00f-17c30e6befcd-maas-api-tls") pod "maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk" (UID: "b172ed58-35b3-4db8-b00f-17c30e6befcd") : secret "maas-api-e2e-inf-b-44546f-serving-cert" not found
Jul 28 12:12:41.587685 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:41.587669    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b172ed58-35b3-4db8-b00f-17c30e6befcd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk\" (UID: \"b172ed58-35b3-4db8-b00f-17c30e6befcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"
Jul 28 12:12:41.595072 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:41.595011    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfw2s\" (UniqueName: \"kubernetes.io/projected/b172ed58-35b3-4db8-b00f-17c30e6befcd-kube-api-access-wfw2s\") pod \"maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk\" (UID: \"b172ed58-35b3-4db8-b00f-17c30e6befcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"
Jul 28 12:12:42.091517 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.091482    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b172ed58-35b3-4db8-b00f-17c30e6befcd-maas-api-tls\") pod \"maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk\" (UID: \"b172ed58-35b3-4db8-b00f-17c30e6befcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"
Jul 28 12:12:42.093907 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.093877    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b172ed58-35b3-4db8-b00f-17c30e6befcd-maas-api-tls\") pod \"maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk\" (UID: \"b172ed58-35b3-4db8-b00f-17c30e6befcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"
Jul 28 12:12:42.123109 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.123083    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd"]
Jul 28 12:12:42.126416 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.126400    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt"]
Jul 28 12:12:42.126530 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.126512    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd"
Jul 28 12:12:42.129080 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.129058    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-44546f\""
Jul 28 12:12:42.129789 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.129769    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt"
Jul 28 12:12:42.145408 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.145384    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt"]
Jul 28 12:12:42.147177 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.147154    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd"]
Jul 28 12:12:42.192384 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.192348    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1445878a-6a8d-46c1-93cb-2d4df75622c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd\" (UID: \"1445878a-6a8d-46c1-93cb-2d4df75622c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd"
Jul 28 12:12:42.192384 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.192388    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnbj4\" (UniqueName: \"kubernetes.io/projected/1445878a-6a8d-46c1-93cb-2d4df75622c2-kube-api-access-bnbj4\") pod \"payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd\" (UID: \"1445878a-6a8d-46c1-93cb-2d4df75622c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd"
Jul 28 12:12:42.192611 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.192412    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt\" (UID: \"1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt"
Jul 28 12:12:42.192611 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.192505    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzpln\" (UniqueName: \"kubernetes.io/projected/1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7-kube-api-access-jzpln\") pod \"payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt\" (UID: \"1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt"
Jul 28 12:12:42.257980 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.257954    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"
Jul 28 12:12:42.293073 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.293039    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1445878a-6a8d-46c1-93cb-2d4df75622c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd\" (UID: \"1445878a-6a8d-46c1-93cb-2d4df75622c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd"
Jul 28 12:12:42.293073 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.293075    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnbj4\" (UniqueName: \"kubernetes.io/projected/1445878a-6a8d-46c1-93cb-2d4df75622c2-kube-api-access-bnbj4\") pod \"payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd\" (UID: \"1445878a-6a8d-46c1-93cb-2d4df75622c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd"
Jul 28 12:12:42.293315 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.293191    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt\" (UID: \"1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt"
Jul 28 12:12:42.293315 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.293290    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzpln\" (UniqueName: \"kubernetes.io/projected/1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7-kube-api-access-jzpln\") pod \"payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt\" (UID: \"1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt"
Jul 28 12:12:42.293813 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.293793    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1445878a-6a8d-46c1-93cb-2d4df75622c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd\" (UID: \"1445878a-6a8d-46c1-93cb-2d4df75622c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd"
Jul 28 12:12:42.293901 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.293879    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt\" (UID: \"1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt"
Jul 28 12:12:42.301931 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.301909    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnbj4\" (UniqueName: \"kubernetes.io/projected/1445878a-6a8d-46c1-93cb-2d4df75622c2-kube-api-access-bnbj4\") pod \"payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd\" (UID: \"1445878a-6a8d-46c1-93cb-2d4df75622c2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd"
Jul 28 12:12:42.302664 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.302644    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzpln\" (UniqueName: \"kubernetes.io/projected/1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7-kube-api-access-jzpln\") pod \"payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt\" (UID: \"1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt"
Jul 28 12:12:42.387690 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.387661    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"]
Jul 28 12:12:42.389470 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:12:42.389440    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb172ed58_35b3_4db8_b00f_17c30e6befcd.slice/crio-518bd8ddecc3b27bb807839be0ab98f475f3ca21eb717056d7dd4ff83e83b754 WatchSource:0}: Error finding container 518bd8ddecc3b27bb807839be0ab98f475f3ca21eb717056d7dd4ff83e83b754: Status 404 returned error can't find the container with id 518bd8ddecc3b27bb807839be0ab98f475f3ca21eb717056d7dd4ff83e83b754
Jul 28 12:12:42.402805 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.402778    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk" event={"ID":"b172ed58-35b3-4db8-b00f-17c30e6befcd","Type":"ContainerStarted","Data":"518bd8ddecc3b27bb807839be0ab98f475f3ca21eb717056d7dd4ff83e83b754"}
Jul 28 12:12:42.438119 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.438093    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd"
Jul 28 12:12:42.444255 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.444237    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt"
Jul 28 12:12:42.575565 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.575537    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd"]
Jul 28 12:12:42.578865 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:12:42.578838    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1445878a_6a8d_46c1_93cb_2d4df75622c2.slice/crio-40df0931676df721af948ba008c9b1d48e5075e8f4ad85202bfd825462ac1b34 WatchSource:0}: Error finding container 40df0931676df721af948ba008c9b1d48e5075e8f4ad85202bfd825462ac1b34: Status 404 returned error can't find the container with id 40df0931676df721af948ba008c9b1d48e5075e8f4ad85202bfd825462ac1b34
Jul 28 12:12:42.802542 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:42.802516    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt"]
Jul 28 12:12:42.804972 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:12:42.804943    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf6a5c4_7b3e_428f_aeec_6d23d6299fc7.slice/crio-1bb4032f3a4f280a6934c6214c30930d7747304b6f13fd66a0133f48381d2559 WatchSource:0}: Error finding container 1bb4032f3a4f280a6934c6214c30930d7747304b6f13fd66a0133f48381d2559: Status 404 returned error can't find the container with id 1bb4032f3a4f280a6934c6214c30930d7747304b6f13fd66a0133f48381d2559
Jul 28 12:12:43.411560 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:43.411522    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt" event={"ID":"1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7","Type":"ContainerStarted","Data":"b153e69a0c1f2dd81808ed8d47360c16c22e4bccb3b143c312f754235d22bf01"}
Jul 28 12:12:43.412008 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:43.411588    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt" event={"ID":"1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7","Type":"ContainerStarted","Data":"1bb4032f3a4f280a6934c6214c30930d7747304b6f13fd66a0133f48381d2559"}
Jul 28 12:12:43.412008 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:43.411704    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt"
Jul 28 12:12:43.413057 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:43.413033    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd" event={"ID":"1445878a-6a8d-46c1-93cb-2d4df75622c2","Type":"ContainerStarted","Data":"2a6300a69a2a9932d4a40f247d53d01ac93c5bb94fcd63f16e9057105b9336c3"}
Jul 28 12:12:43.413176 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:43.413077    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd" event={"ID":"1445878a-6a8d-46c1-93cb-2d4df75622c2","Type":"ContainerStarted","Data":"40df0931676df721af948ba008c9b1d48e5075e8f4ad85202bfd825462ac1b34"}
Jul 28 12:12:43.413176 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:43.413138    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd"
Jul 28 12:12:43.414412 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:43.414393    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk" event={"ID":"b172ed58-35b3-4db8-b00f-17c30e6befcd","Type":"ContainerStarted","Data":"d7af83a8b56c1c554694d0e99e01ffb332749ac2f0c60226a5e410a573e1ccb8"}
Jul 28 12:12:43.414521 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:43.414505    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"
Jul 28 12:12:43.427709 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:43.427672    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt" podStartSLOduration=1.427661533 podStartE2EDuration="1.427661533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:12:43.42649098 +0000 UTC m=+2515.285146982" watchObservedRunningTime="2026-07-28 12:12:43.427661533 +0000 UTC m=+2515.286317580"
Jul 28 12:12:43.443198 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:43.443161    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd" podStartSLOduration=1.443149062 podStartE2EDuration="1.443149062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:12:43.442007189 +0000 UTC m=+2515.300663191" watchObservedRunningTime="2026-07-28 12:12:43.443149062 +0000 UTC m=+2515.301805038"
Jul 28 12:12:43.460201 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:43.460164    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk" podStartSLOduration=2.10775902 podStartE2EDuration="2.460153068s" totalImagesPullingTime="352.394048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:12:43.458940601 +0000 UTC m=+2515.317596600" watchObservedRunningTime="2026-07-28 12:12:43.460153068 +0000 UTC m=+2515.318809045"
Jul 28 12:12:46.374512 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:46.374481    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn"
Jul 28 12:12:46.374963 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:46.374944    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq"
Jul 28 12:12:48.744148 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:48.744119    2561 scope.go:117] "RemoveContainer" containerID="c98cc3e687aa5b2d81c6919af0a16a4f24bf305821241778e681cd1af8e1b42b"
Jul 28 12:12:49.028814 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:49.028723    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:12:49.426116 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:49.426087    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"
Jul 28 12:12:54.422226 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:54.422196    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd"
Jul 28 12:12:54.422651 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:12:54.422265    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt"
Jul 28 12:13:03.623119 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:03.623085    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:13:19.402696 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.402664    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"]
Jul 28 12:13:19.407336 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.407304    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:19.410285 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.409996    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-44546f\"/\"openshift-service-ca.crt\""
Jul 28 12:13:19.410285 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.410047    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-44546f\"/\"default-dockercfg-sw5pz\""
Jul 28 12:13:19.410285 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.410049    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-44546f\"/\"test-model-44546f-kserve-self-signed-certs\""
Jul 28 12:13:19.410285 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.410196    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-44546f\"/\"kube-root-ca.crt\""
Jul 28 12:13:19.417545 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.417519    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"]
Jul 28 12:13:19.504481 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.504449    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-model-cache\") pod \"test-model-44546f-kserve-c5c59b755-znzj6\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") " pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:19.504676 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.504493    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dswgs\" (UniqueName: \"kubernetes.io/projected/1f6679a4-edc3-46a7-b9b0-f60ff9219392-kube-api-access-dswgs\") pod \"test-model-44546f-kserve-c5c59b755-znzj6\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") " pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:19.504676 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.504614    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-home\") pod \"test-model-44546f-kserve-c5c59b755-znzj6\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") " pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:19.504676 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.504639    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-dshm\") pod \"test-model-44546f-kserve-c5c59b755-znzj6\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") " pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:19.504676 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.504659    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f6679a4-edc3-46a7-b9b0-f60ff9219392-tls-certs\") pod \"test-model-44546f-kserve-c5c59b755-znzj6\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") " pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:19.504857 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.504755    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-tmp-dir\") pod \"test-model-44546f-kserve-c5c59b755-znzj6\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") " pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:19.605835 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.605802    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-home\") pod \"test-model-44546f-kserve-c5c59b755-znzj6\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") " pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:19.605835 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.605836    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-dshm\") pod \"test-model-44546f-kserve-c5c59b755-znzj6\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") " pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:19.606014 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.605851    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f6679a4-edc3-46a7-b9b0-f60ff9219392-tls-certs\") pod \"test-model-44546f-kserve-c5c59b755-znzj6\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") " pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:19.606014 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.605893    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-tmp-dir\") pod \"test-model-44546f-kserve-c5c59b755-znzj6\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") " pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:19.606014 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.605971    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-model-cache\") pod \"test-model-44546f-kserve-c5c59b755-znzj6\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") " pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:19.606014 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.606005    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dswgs\" (UniqueName: \"kubernetes.io/projected/1f6679a4-edc3-46a7-b9b0-f60ff9219392-kube-api-access-dswgs\") pod \"test-model-44546f-kserve-c5c59b755-znzj6\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") " pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:19.606310 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.606286    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-home\") pod \"test-model-44546f-kserve-c5c59b755-znzj6\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") " pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:19.606481 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.606462    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-tmp-dir\") pod \"test-model-44546f-kserve-c5c59b755-znzj6\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") " pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:19.606603 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.606506    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-model-cache\") pod \"test-model-44546f-kserve-c5c59b755-znzj6\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") " pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:19.607963 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.607938    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-dshm\") pod \"test-model-44546f-kserve-c5c59b755-znzj6\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") " pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:19.608378 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.608362    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f6679a4-edc3-46a7-b9b0-f60ff9219392-tls-certs\") pod \"test-model-44546f-kserve-c5c59b755-znzj6\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") " pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:19.613952 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.613934    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dswgs\" (UniqueName: \"kubernetes.io/projected/1f6679a4-edc3-46a7-b9b0-f60ff9219392-kube-api-access-dswgs\") pod \"test-model-44546f-kserve-c5c59b755-znzj6\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") " pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:19.718406 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.718346    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:19.846913 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:19.846891    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"]
Jul 28 12:13:19.849396 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:13:19.849373    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f6679a4_edc3_46a7_b9b0_f60ff9219392.slice/crio-6d4a4a9f027a99e1f053d20ee1eac793e54945bdd03b6d5fc917eabf7a73b280 WatchSource:0}: Error finding container 6d4a4a9f027a99e1f053d20ee1eac793e54945bdd03b6d5fc917eabf7a73b280: Status 404 returned error can't find the container with id 6d4a4a9f027a99e1f053d20ee1eac793e54945bdd03b6d5fc917eabf7a73b280
Jul 28 12:13:20.583983 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:20.583944    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6" event={"ID":"1f6679a4-edc3-46a7-b9b0-f60ff9219392","Type":"ContainerStarted","Data":"f56b1d3699ebcd295b05b0c351e240c05350e546f51d3b805bc5e8b051ee2e0e"}
Jul 28 12:13:20.583983 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:20.583985    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6" event={"ID":"1f6679a4-edc3-46a7-b9b0-f60ff9219392","Type":"ContainerStarted","Data":"6d4a4a9f027a99e1f053d20ee1eac793e54945bdd03b6d5fc917eabf7a73b280"}
Jul 28 12:13:20.599624 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:20.599561    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6" podStartSLOduration=1.599547502 podStartE2EDuration="1.599547502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:13:20.598731978 +0000 UTC m=+2552.457387976" watchObservedRunningTime="2026-07-28 12:13:20.599547502 +0000 UTC m=+2552.458203500"
Jul 28 12:13:23.527401 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:23.527368    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:13:29.719244 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:29.719208    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:29.719823 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:29.719284    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:29.731792 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:29.731767    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:30.637162 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:30.637127    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:13:37.237972 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:13:37.237935    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:14:01.476785 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:01.476749    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"]
Jul 28 12:14:05.243342 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:05.243310    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:14:12.162779 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.162705    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"]
Jul 28 12:14:12.163203 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.162944    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7" podUID="33087b5b-9f07-46da-9564-9f79dcbf4813" containerName="maas-api" containerID="cri-o://c1b5a0b1f8065bd30753e0fdbd5608ee90f29ec86a043464531f7fc57c9e8193" gracePeriod=30
Jul 28 12:14:12.220399 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.220369    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn"]
Jul 28 12:14:12.220622 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.220595    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn" podUID="936cd400-18a9-421f-900f-d026de89a7bc" containerName="payload-processing" containerID="cri-o://06fc64694a7bcfde6c23e7de7abb52bba14b15f03d4ac180fe30424fb2cf4a66" gracePeriod=30
Jul 28 12:14:12.227504 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.227474    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq"]
Jul 28 12:14:12.227736 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.227702    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq" podUID="e96f10b6-2b18-404f-bb89-6ca66fa49533" containerName="payload-pre-processing" containerID="cri-o://b427d052653f929db79a6f66b5d10af9ab64499e1c518760e4625b9d08dd9b30" gracePeriod=30
Jul 28 12:14:12.482244 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.482219    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"
Jul 28 12:14:12.508790 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.508764    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn"
Jul 28 12:14:12.516967 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.516946    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq"
Jul 28 12:14:12.546846 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.546824    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33087b5b-9f07-46da-9564-9f79dcbf4813-maas-api-tls\") pod \"33087b5b-9f07-46da-9564-9f79dcbf4813\" (UID: \"33087b5b-9f07-46da-9564-9f79dcbf4813\") "
Jul 28 12:14:12.546993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.546873    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936cd400-18a9-421f-900f-d026de89a7bc-plugins-config-volume\") pod \"936cd400-18a9-421f-900f-d026de89a7bc\" (UID: \"936cd400-18a9-421f-900f-d026de89a7bc\") "
Jul 28 12:14:12.546993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.546902    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brcj5\" (UniqueName: \"kubernetes.io/projected/33087b5b-9f07-46da-9564-9f79dcbf4813-kube-api-access-brcj5\") pod \"33087b5b-9f07-46da-9564-9f79dcbf4813\" (UID: \"33087b5b-9f07-46da-9564-9f79dcbf4813\") "
Jul 28 12:14:12.546993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.546935    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96f10b6-2b18-404f-bb89-6ca66fa49533-plugins-config-volume\") pod \"e96f10b6-2b18-404f-bb89-6ca66fa49533\" (UID: \"e96f10b6-2b18-404f-bb89-6ca66fa49533\") "
Jul 28 12:14:12.546993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.546987    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33087b5b-9f07-46da-9564-9f79dcbf4813-openshift-service-ca\") pod \"33087b5b-9f07-46da-9564-9f79dcbf4813\" (UID: \"33087b5b-9f07-46da-9564-9f79dcbf4813\") "
Jul 28 12:14:12.547215 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.547012    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf7l2\" (UniqueName: \"kubernetes.io/projected/e96f10b6-2b18-404f-bb89-6ca66fa49533-kube-api-access-xf7l2\") pod \"e96f10b6-2b18-404f-bb89-6ca66fa49533\" (UID: \"e96f10b6-2b18-404f-bb89-6ca66fa49533\") "
Jul 28 12:14:12.547215 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.547050    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhfjp\" (UniqueName: \"kubernetes.io/projected/936cd400-18a9-421f-900f-d026de89a7bc-kube-api-access-zhfjp\") pod \"936cd400-18a9-421f-900f-d026de89a7bc\" (UID: \"936cd400-18a9-421f-900f-d026de89a7bc\") "
Jul 28 12:14:12.547332 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.547280    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936cd400-18a9-421f-900f-d026de89a7bc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "936cd400-18a9-421f-900f-d026de89a7bc" (UID: "936cd400-18a9-421f-900f-d026de89a7bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:14:12.547712 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.547506    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e96f10b6-2b18-404f-bb89-6ca66fa49533-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e96f10b6-2b18-404f-bb89-6ca66fa49533" (UID: "e96f10b6-2b18-404f-bb89-6ca66fa49533"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:14:12.547712 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.547697    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33087b5b-9f07-46da-9564-9f79dcbf4813-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33087b5b-9f07-46da-9564-9f79dcbf4813" (UID: "33087b5b-9f07-46da-9564-9f79dcbf4813"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:14:12.549099 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.549078    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33087b5b-9f07-46da-9564-9f79dcbf4813-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33087b5b-9f07-46da-9564-9f79dcbf4813" (UID: "33087b5b-9f07-46da-9564-9f79dcbf4813"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:14:12.549276 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.549256    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936cd400-18a9-421f-900f-d026de89a7bc-kube-api-access-zhfjp" (OuterVolumeSpecName: "kube-api-access-zhfjp") pod "936cd400-18a9-421f-900f-d026de89a7bc" (UID: "936cd400-18a9-421f-900f-d026de89a7bc"). InnerVolumeSpecName "kube-api-access-zhfjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:14:12.549414 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.549399    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33087b5b-9f07-46da-9564-9f79dcbf4813-kube-api-access-brcj5" (OuterVolumeSpecName: "kube-api-access-brcj5") pod "33087b5b-9f07-46da-9564-9f79dcbf4813" (UID: "33087b5b-9f07-46da-9564-9f79dcbf4813"). InnerVolumeSpecName "kube-api-access-brcj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:14:12.549490 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.549476    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96f10b6-2b18-404f-bb89-6ca66fa49533-kube-api-access-xf7l2" (OuterVolumeSpecName: "kube-api-access-xf7l2") pod "e96f10b6-2b18-404f-bb89-6ca66fa49533" (UID: "e96f10b6-2b18-404f-bb89-6ca66fa49533"). InnerVolumeSpecName "kube-api-access-xf7l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:14:12.648746 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.648716    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936cd400-18a9-421f-900f-d026de89a7bc-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:12.648746 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.648741    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brcj5\" (UniqueName: \"kubernetes.io/projected/33087b5b-9f07-46da-9564-9f79dcbf4813-kube-api-access-brcj5\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:12.648746 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.648752    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96f10b6-2b18-404f-bb89-6ca66fa49533-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:12.648968 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.648760    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33087b5b-9f07-46da-9564-9f79dcbf4813-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:12.648968 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.648770    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf7l2\" (UniqueName: \"kubernetes.io/projected/e96f10b6-2b18-404f-bb89-6ca66fa49533-kube-api-access-xf7l2\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:12.648968 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.648780    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhfjp\" (UniqueName: \"kubernetes.io/projected/936cd400-18a9-421f-900f-d026de89a7bc-kube-api-access-zhfjp\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:12.648968 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.648788    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33087b5b-9f07-46da-9564-9f79dcbf4813-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:12.812217 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.812182    2561 generic.go:356] "Generic (PLEG): container finished" podID="936cd400-18a9-421f-900f-d026de89a7bc" containerID="06fc64694a7bcfde6c23e7de7abb52bba14b15f03d4ac180fe30424fb2cf4a66" exitCode=0
Jul 28 12:14:12.812388 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.812228    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn"
Jul 28 12:14:12.812388 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.812263    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn" event={"ID":"936cd400-18a9-421f-900f-d026de89a7bc","Type":"ContainerDied","Data":"06fc64694a7bcfde6c23e7de7abb52bba14b15f03d4ac180fe30424fb2cf4a66"}
Jul 28 12:14:12.812388 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.812328    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn" event={"ID":"936cd400-18a9-421f-900f-d026de89a7bc","Type":"ContainerDied","Data":"71492131c19b7ee290a7f549719d35a1ec47f8367258680ae9a21f7d147cc3a1"}
Jul 28 12:14:12.812388 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.812361    2561 scope.go:117] "RemoveContainer" containerID="06fc64694a7bcfde6c23e7de7abb52bba14b15f03d4ac180fe30424fb2cf4a66"
Jul 28 12:14:12.813636 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.813612    2561 generic.go:356] "Generic (PLEG): container finished" podID="e96f10b6-2b18-404f-bb89-6ca66fa49533" containerID="b427d052653f929db79a6f66b5d10af9ab64499e1c518760e4625b9d08dd9b30" exitCode=0
Jul 28 12:14:12.813775 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.813653    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq"
Jul 28 12:14:12.813775 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.813690    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq" event={"ID":"e96f10b6-2b18-404f-bb89-6ca66fa49533","Type":"ContainerDied","Data":"b427d052653f929db79a6f66b5d10af9ab64499e1c518760e4625b9d08dd9b30"}
Jul 28 12:14:12.813775 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.813729    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq" event={"ID":"e96f10b6-2b18-404f-bb89-6ca66fa49533","Type":"ContainerDied","Data":"4a8c68b199de34f890f552060748f7318eb89a7b289cadd558491a699e90a36c"}
Jul 28 12:14:12.815090 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.815066    2561 generic.go:356] "Generic (PLEG): container finished" podID="33087b5b-9f07-46da-9564-9f79dcbf4813" containerID="c1b5a0b1f8065bd30753e0fdbd5608ee90f29ec86a043464531f7fc57c9e8193" exitCode=0
Jul 28 12:14:12.815185 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.815105    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"
Jul 28 12:14:12.815185 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.815106    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7" event={"ID":"33087b5b-9f07-46da-9564-9f79dcbf4813","Type":"ContainerDied","Data":"c1b5a0b1f8065bd30753e0fdbd5608ee90f29ec86a043464531f7fc57c9e8193"}
Jul 28 12:14:12.815185 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.815147    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7" event={"ID":"33087b5b-9f07-46da-9564-9f79dcbf4813","Type":"ContainerDied","Data":"99140b0230748bbaae2b0f1d965ec08867292cdd5e55d6d05d91fddd09d3a28b"}
Jul 28 12:14:12.822380 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.822366    2561 scope.go:117] "RemoveContainer" containerID="06fc64694a7bcfde6c23e7de7abb52bba14b15f03d4ac180fe30424fb2cf4a66"
Jul 28 12:14:12.822662 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:14:12.822643    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06fc64694a7bcfde6c23e7de7abb52bba14b15f03d4ac180fe30424fb2cf4a66\": container with ID starting with 06fc64694a7bcfde6c23e7de7abb52bba14b15f03d4ac180fe30424fb2cf4a66 not found: ID does not exist" containerID="06fc64694a7bcfde6c23e7de7abb52bba14b15f03d4ac180fe30424fb2cf4a66"
Jul 28 12:14:12.822718 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.822669    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06fc64694a7bcfde6c23e7de7abb52bba14b15f03d4ac180fe30424fb2cf4a66"} err="failed to get container status \"06fc64694a7bcfde6c23e7de7abb52bba14b15f03d4ac180fe30424fb2cf4a66\": rpc error: code = NotFound desc = could not find container \"06fc64694a7bcfde6c23e7de7abb52bba14b15f03d4ac180fe30424fb2cf4a66\": container with ID starting with 06fc64694a7bcfde6c23e7de7abb52bba14b15f03d4ac180fe30424fb2cf4a66 not found: ID does not exist"
Jul 28 12:14:12.822718 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.822685    2561 scope.go:117] "RemoveContainer" containerID="b427d052653f929db79a6f66b5d10af9ab64499e1c518760e4625b9d08dd9b30"
Jul 28 12:14:12.829973 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.829954    2561 scope.go:117] "RemoveContainer" containerID="b427d052653f929db79a6f66b5d10af9ab64499e1c518760e4625b9d08dd9b30"
Jul 28 12:14:12.830972 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:14:12.830833    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b427d052653f929db79a6f66b5d10af9ab64499e1c518760e4625b9d08dd9b30\": container with ID starting with b427d052653f929db79a6f66b5d10af9ab64499e1c518760e4625b9d08dd9b30 not found: ID does not exist" containerID="b427d052653f929db79a6f66b5d10af9ab64499e1c518760e4625b9d08dd9b30"
Jul 28 12:14:12.830972 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.830864    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b427d052653f929db79a6f66b5d10af9ab64499e1c518760e4625b9d08dd9b30"} err="failed to get container status \"b427d052653f929db79a6f66b5d10af9ab64499e1c518760e4625b9d08dd9b30\": rpc error: code = NotFound desc = could not find container \"b427d052653f929db79a6f66b5d10af9ab64499e1c518760e4625b9d08dd9b30\": container with ID starting with b427d052653f929db79a6f66b5d10af9ab64499e1c518760e4625b9d08dd9b30 not found: ID does not exist"
Jul 28 12:14:12.830972 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.830901    2561 scope.go:117] "RemoveContainer" containerID="c1b5a0b1f8065bd30753e0fdbd5608ee90f29ec86a043464531f7fc57c9e8193"
Jul 28 12:14:12.832420 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.832402    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"]
Jul 28 12:14:12.836039 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.836019    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-21898b-6d5cc67f49-5rgg7"]
Jul 28 12:14:12.838313 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.838296    2561 scope.go:117] "RemoveContainer" containerID="c1b5a0b1f8065bd30753e0fdbd5608ee90f29ec86a043464531f7fc57c9e8193"
Jul 28 12:14:12.838560 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:14:12.838531    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b5a0b1f8065bd30753e0fdbd5608ee90f29ec86a043464531f7fc57c9e8193\": container with ID starting with c1b5a0b1f8065bd30753e0fdbd5608ee90f29ec86a043464531f7fc57c9e8193 not found: ID does not exist" containerID="c1b5a0b1f8065bd30753e0fdbd5608ee90f29ec86a043464531f7fc57c9e8193"
Jul 28 12:14:12.838622 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.838565    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b5a0b1f8065bd30753e0fdbd5608ee90f29ec86a043464531f7fc57c9e8193"} err="failed to get container status \"c1b5a0b1f8065bd30753e0fdbd5608ee90f29ec86a043464531f7fc57c9e8193\": rpc error: code = NotFound desc = could not find container \"c1b5a0b1f8065bd30753e0fdbd5608ee90f29ec86a043464531f7fc57c9e8193\": container with ID starting with c1b5a0b1f8065bd30753e0fdbd5608ee90f29ec86a043464531f7fc57c9e8193 not found: ID does not exist"
Jul 28 12:14:12.844663 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.844643    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq"]
Jul 28 12:14:12.848356 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.848337    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-21898b-6bdbcc56b4-c5phq"]
Jul 28 12:14:12.858101 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.858082    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn"]
Jul 28 12:14:12.860054 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:12.860036    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-21898b-6f8b9879cd-vqnfn"]
Jul 28 12:14:14.546730 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:14.546700    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33087b5b-9f07-46da-9564-9f79dcbf4813" path="/var/lib/kubelet/pods/33087b5b-9f07-46da-9564-9f79dcbf4813/volumes"
Jul 28 12:14:14.547100 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:14.547048    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="936cd400-18a9-421f-900f-d026de89a7bc" path="/var/lib/kubelet/pods/936cd400-18a9-421f-900f-d026de89a7bc/volumes"
Jul 28 12:14:14.547365 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:14.547352    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e96f10b6-2b18-404f-bb89-6ca66fa49533" path="/var/lib/kubelet/pods/e96f10b6-2b18-404f-bb89-6ca66fa49533/volumes"
Jul 28 12:14:16.496655 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.496609    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6" podUID="1f6679a4-edc3-46a7-b9b0-f60ff9219392" containerName="main" containerID="cri-o://f56b1d3699ebcd295b05b0c351e240c05350e546f51d3b805bc5e8b051ee2e0e" gracePeriod=45
Jul 28 12:14:16.733772 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.733751    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:14:16.788836 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.788758    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dswgs\" (UniqueName: \"kubernetes.io/projected/1f6679a4-edc3-46a7-b9b0-f60ff9219392-kube-api-access-dswgs\") pod \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") "
Jul 28 12:14:16.788836 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.788818    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-home\") pod \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") "
Jul 28 12:14:16.789023 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.788845    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-dshm\") pod \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") "
Jul 28 12:14:16.789023 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.788915    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-model-cache\") pod \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") "
Jul 28 12:14:16.789023 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.788961    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-tmp-dir\") pod \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") "
Jul 28 12:14:16.789188 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.789021    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f6679a4-edc3-46a7-b9b0-f60ff9219392-tls-certs\") pod \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\" (UID: \"1f6679a4-edc3-46a7-b9b0-f60ff9219392\") "
Jul 28 12:14:16.789188 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.789029    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-home" (OuterVolumeSpecName: "home") pod "1f6679a4-edc3-46a7-b9b0-f60ff9219392" (UID: "1f6679a4-edc3-46a7-b9b0-f60ff9219392"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:14:16.789274 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.789200    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-model-cache" (OuterVolumeSpecName: "model-cache") pod "1f6679a4-edc3-46a7-b9b0-f60ff9219392" (UID: "1f6679a4-edc3-46a7-b9b0-f60ff9219392"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:14:16.789274 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.789221    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1f6679a4-edc3-46a7-b9b0-f60ff9219392" (UID: "1f6679a4-edc3-46a7-b9b0-f60ff9219392"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:14:16.789362 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.789283    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-model-cache\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:16.789362 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.789298    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-tmp-dir\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:16.789362 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.789311    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-home\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:16.791083 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.791060    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f6679a4-edc3-46a7-b9b0-f60ff9219392-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1f6679a4-edc3-46a7-b9b0-f60ff9219392" (UID: "1f6679a4-edc3-46a7-b9b0-f60ff9219392"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:14:16.791285 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.791269    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-dshm" (OuterVolumeSpecName: "dshm") pod "1f6679a4-edc3-46a7-b9b0-f60ff9219392" (UID: "1f6679a4-edc3-46a7-b9b0-f60ff9219392"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:14:16.791363 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.791329    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6679a4-edc3-46a7-b9b0-f60ff9219392-kube-api-access-dswgs" (OuterVolumeSpecName: "kube-api-access-dswgs") pod "1f6679a4-edc3-46a7-b9b0-f60ff9219392" (UID: "1f6679a4-edc3-46a7-b9b0-f60ff9219392"). InnerVolumeSpecName "kube-api-access-dswgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:14:16.836453 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.836421    2561 generic.go:356] "Generic (PLEG): container finished" podID="1f6679a4-edc3-46a7-b9b0-f60ff9219392" containerID="f56b1d3699ebcd295b05b0c351e240c05350e546f51d3b805bc5e8b051ee2e0e" exitCode=0
Jul 28 12:14:16.836602 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.836461    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"
Jul 28 12:14:16.836602 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.836504    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6" event={"ID":"1f6679a4-edc3-46a7-b9b0-f60ff9219392","Type":"ContainerDied","Data":"f56b1d3699ebcd295b05b0c351e240c05350e546f51d3b805bc5e8b051ee2e0e"}
Jul 28 12:14:16.836602 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.836545    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6" event={"ID":"1f6679a4-edc3-46a7-b9b0-f60ff9219392","Type":"ContainerDied","Data":"6d4a4a9f027a99e1f053d20ee1eac793e54945bdd03b6d5fc917eabf7a73b280"}
Jul 28 12:14:16.836602 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.836565    2561 scope.go:117] "RemoveContainer" containerID="f56b1d3699ebcd295b05b0c351e240c05350e546f51d3b805bc5e8b051ee2e0e"
Jul 28 12:14:16.845502 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.845487    2561 scope.go:117] "RemoveContainer" containerID="f56b1d3699ebcd295b05b0c351e240c05350e546f51d3b805bc5e8b051ee2e0e"
Jul 28 12:14:16.845764 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:14:16.845742    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56b1d3699ebcd295b05b0c351e240c05350e546f51d3b805bc5e8b051ee2e0e\": container with ID starting with f56b1d3699ebcd295b05b0c351e240c05350e546f51d3b805bc5e8b051ee2e0e not found: ID does not exist" containerID="f56b1d3699ebcd295b05b0c351e240c05350e546f51d3b805bc5e8b051ee2e0e"
Jul 28 12:14:16.845886 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.845772    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56b1d3699ebcd295b05b0c351e240c05350e546f51d3b805bc5e8b051ee2e0e"} err="failed to get container status \"f56b1d3699ebcd295b05b0c351e240c05350e546f51d3b805bc5e8b051ee2e0e\": rpc error: code = NotFound desc = could not find container \"f56b1d3699ebcd295b05b0c351e240c05350e546f51d3b805bc5e8b051ee2e0e\": container with ID starting with f56b1d3699ebcd295b05b0c351e240c05350e546f51d3b805bc5e8b051ee2e0e not found: ID does not exist"
Jul 28 12:14:16.858120 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.858094    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"]
Jul 28 12:14:16.862026 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.862004    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-44546f/test-model-44546f-kserve-c5c59b755-znzj6"]
Jul 28 12:14:16.890716 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.890692    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f6679a4-edc3-46a7-b9b0-f60ff9219392-dshm\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:16.890716 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.890715    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f6679a4-edc3-46a7-b9b0-f60ff9219392-tls-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:16.890843 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:16.890726    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dswgs\" (UniqueName: \"kubernetes.io/projected/1f6679a4-edc3-46a7-b9b0-f60ff9219392-kube-api-access-dswgs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:17.339191 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:17.339161    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:14:18.546429 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:18.546394    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f6679a4-edc3-46a7-b9b0-f60ff9219392" path="/var/lib/kubelet/pods/1f6679a4-edc3-46a7-b9b0-f60ff9219392/volumes"
Jul 28 12:14:24.044608 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.044566    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44546f-5dqpt"]
Jul 28 12:14:24.044985 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.044903    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33087b5b-9f07-46da-9564-9f79dcbf4813" containerName="maas-api"
Jul 28 12:14:24.044985 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.044915    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="33087b5b-9f07-46da-9564-9f79dcbf4813" containerName="maas-api"
Jul 28 12:14:24.044985 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.044933    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f6679a4-edc3-46a7-b9b0-f60ff9219392" containerName="main"
Jul 28 12:14:24.044985 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.044941    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f6679a4-edc3-46a7-b9b0-f60ff9219392" containerName="main"
Jul 28 12:14:24.044985 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.044953    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e96f10b6-2b18-404f-bb89-6ca66fa49533" containerName="payload-pre-processing"
Jul 28 12:14:24.044985 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.044959    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e96f10b6-2b18-404f-bb89-6ca66fa49533" containerName="payload-pre-processing"
Jul 28 12:14:24.044985 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.044966    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="936cd400-18a9-421f-900f-d026de89a7bc" containerName="payload-processing"
Jul 28 12:14:24.044985 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.044972    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="936cd400-18a9-421f-900f-d026de89a7bc" containerName="payload-processing"
Jul 28 12:14:24.045256 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.045043    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e96f10b6-2b18-404f-bb89-6ca66fa49533" containerName="payload-pre-processing"
Jul 28 12:14:24.045256 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.045052    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="936cd400-18a9-421f-900f-d026de89a7bc" containerName="payload-processing"
Jul 28 12:14:24.045256 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.045060    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="33087b5b-9f07-46da-9564-9f79dcbf4813" containerName="maas-api"
Jul 28 12:14:24.045256 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.045067    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f6679a4-edc3-46a7-b9b0-f60ff9219392" containerName="main"
Jul 28 12:14:24.049274 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.049260    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44546f-5dqpt"
Jul 28 12:14:24.051485 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.051463    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ts4x9\""
Jul 28 12:14:24.055036 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.055012    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44546f-5dqpt"]
Jul 28 12:14:24.150887 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.150861    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d08cf0-6f63-4635-a9d8-78463ddf8ba4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-44546f-5dqpt\" (UID: \"48d08cf0-6f63-4635-a9d8-78463ddf8ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44546f-5dqpt"
Jul 28 12:14:24.252200 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.252168    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d08cf0-6f63-4635-a9d8-78463ddf8ba4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-44546f-5dqpt\" (UID: \"48d08cf0-6f63-4635-a9d8-78463ddf8ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44546f-5dqpt"
Jul 28 12:14:24.252809 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.252788    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d08cf0-6f63-4635-a9d8-78463ddf8ba4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-44546f-5dqpt\" (UID: \"48d08cf0-6f63-4635-a9d8-78463ddf8ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44546f-5dqpt"
Jul 28 12:14:24.360979 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.360950    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44546f-5dqpt"
Jul 28 12:14:24.490073 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.490030    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44546f-5dqpt"]
Jul 28 12:14:24.492903 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:14:24.492878    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d08cf0_6f63_4635_a9d8_78463ddf8ba4.slice/crio-7a6e4e812f11a4b0fef8b5406db132528c93c7fbec623114eabea5dde423474b WatchSource:0}: Error finding container 7a6e4e812f11a4b0fef8b5406db132528c93c7fbec623114eabea5dde423474b: Status 404 returned error can't find the container with id 7a6e4e812f11a4b0fef8b5406db132528c93c7fbec623114eabea5dde423474b
Jul 28 12:14:24.872933 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.872893    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44546f-5dqpt" event={"ID":"48d08cf0-6f63-4635-a9d8-78463ddf8ba4","Type":"ContainerStarted","Data":"5a86c8346f180d0569b55b3bc50e121c81d9939fd3cf38cc21c2b2da45503b90"}
Jul 28 12:14:24.873082 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.872952    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44546f-5dqpt" event={"ID":"48d08cf0-6f63-4635-a9d8-78463ddf8ba4","Type":"ContainerStarted","Data":"7a6e4e812f11a4b0fef8b5406db132528c93c7fbec623114eabea5dde423474b"}
Jul 28 12:14:24.888389 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:24.888344    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44546f-5dqpt" podStartSLOduration=0.888328859 podStartE2EDuration="888.328859ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:14:24.887748928 +0000 UTC m=+2616.746404929" watchObservedRunningTime="2026-07-28 12:14:24.888328859 +0000 UTC m=+2616.746984905"
Jul 28 12:14:25.879279 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:25.879243    2561 generic.go:356] "Generic (PLEG): container finished" podID="48d08cf0-6f63-4635-a9d8-78463ddf8ba4" containerID="5a86c8346f180d0569b55b3bc50e121c81d9939fd3cf38cc21c2b2da45503b90" exitCode=0
Jul 28 12:14:25.879728 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:25.879297    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44546f-5dqpt" event={"ID":"48d08cf0-6f63-4635-a9d8-78463ddf8ba4","Type":"ContainerDied","Data":"5a86c8346f180d0569b55b3bc50e121c81d9939fd3cf38cc21c2b2da45503b90"}
Jul 28 12:14:27.013692 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:27.013664    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44546f-5dqpt"
Jul 28 12:14:27.080139 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:27.080113    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d08cf0-6f63-4635-a9d8-78463ddf8ba4-maas-api-service-ca\") pod \"48d08cf0-6f63-4635-a9d8-78463ddf8ba4\" (UID: \"48d08cf0-6f63-4635-a9d8-78463ddf8ba4\") "
Jul 28 12:14:27.080451 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:27.080430    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d08cf0-6f63-4635-a9d8-78463ddf8ba4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48d08cf0-6f63-4635-a9d8-78463ddf8ba4" (UID: "48d08cf0-6f63-4635-a9d8-78463ddf8ba4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:14:27.181017 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:27.180946    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48d08cf0-6f63-4635-a9d8-78463ddf8ba4-maas-api-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:27.889279 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:27.889250    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44546f-5dqpt"
Jul 28 12:14:27.889450 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:27.889297    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44546f-5dqpt" event={"ID":"48d08cf0-6f63-4635-a9d8-78463ddf8ba4","Type":"ContainerDied","Data":"7a6e4e812f11a4b0fef8b5406db132528c93c7fbec623114eabea5dde423474b"}
Jul 28 12:14:27.889450 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:27.889342    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a6e4e812f11a4b0fef8b5406db132528c93c7fbec623114eabea5dde423474b"
Jul 28 12:14:34.515402 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.515369    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"]
Jul 28 12:14:34.515900 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.515562    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk" podUID="b172ed58-35b3-4db8-b00f-17c30e6befcd" containerName="maas-api" containerID="cri-o://d7af83a8b56c1c554694d0e99e01ffb332749ac2f0c60226a5e410a573e1ccb8" gracePeriod=30
Jul 28 12:14:34.571350 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.571252    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt"]
Jul 28 12:14:34.571557 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.571525    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt" podUID="1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7" containerName="payload-processing" containerID="cri-o://b153e69a0c1f2dd81808ed8d47360c16c22e4bccb3b143c312f754235d22bf01" gracePeriod=30
Jul 28 12:14:34.580176 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.580146    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd"]
Jul 28 12:14:34.580413 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.580367    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd" podUID="1445878a-6a8d-46c1-93cb-2d4df75622c2" containerName="payload-pre-processing" containerID="cri-o://2a6300a69a2a9932d4a40f247d53d01ac93c5bb94fcd63f16e9057105b9336c3" gracePeriod=30
Jul 28 12:14:34.823644 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.823616    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"
Jul 28 12:14:34.868420 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.868394    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt"
Jul 28 12:14:34.871381 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.871360    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd"
Jul 28 12:14:34.922722 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.922690    2561 generic.go:356] "Generic (PLEG): container finished" podID="1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7" containerID="b153e69a0c1f2dd81808ed8d47360c16c22e4bccb3b143c312f754235d22bf01" exitCode=0
Jul 28 12:14:34.922912 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.922740    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt"
Jul 28 12:14:34.922912 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.922782    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt" event={"ID":"1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7","Type":"ContainerDied","Data":"b153e69a0c1f2dd81808ed8d47360c16c22e4bccb3b143c312f754235d22bf01"}
Jul 28 12:14:34.922912 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.922839    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt" event={"ID":"1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7","Type":"ContainerDied","Data":"1bb4032f3a4f280a6934c6214c30930d7747304b6f13fd66a0133f48381d2559"}
Jul 28 12:14:34.922912 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.922870    2561 scope.go:117] "RemoveContainer" containerID="b153e69a0c1f2dd81808ed8d47360c16c22e4bccb3b143c312f754235d22bf01"
Jul 28 12:14:34.924003 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.923982    2561 generic.go:356] "Generic (PLEG): container finished" podID="1445878a-6a8d-46c1-93cb-2d4df75622c2" containerID="2a6300a69a2a9932d4a40f247d53d01ac93c5bb94fcd63f16e9057105b9336c3" exitCode=0
Jul 28 12:14:34.924099 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.924027    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd"
Jul 28 12:14:34.924099 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.924066    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd" event={"ID":"1445878a-6a8d-46c1-93cb-2d4df75622c2","Type":"ContainerDied","Data":"2a6300a69a2a9932d4a40f247d53d01ac93c5bb94fcd63f16e9057105b9336c3"}
Jul 28 12:14:34.924214 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.924123    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd" event={"ID":"1445878a-6a8d-46c1-93cb-2d4df75622c2","Type":"ContainerDied","Data":"40df0931676df721af948ba008c9b1d48e5075e8f4ad85202bfd825462ac1b34"}
Jul 28 12:14:34.925187 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.925168    2561 generic.go:356] "Generic (PLEG): container finished" podID="b172ed58-35b3-4db8-b00f-17c30e6befcd" containerID="d7af83a8b56c1c554694d0e99e01ffb332749ac2f0c60226a5e410a573e1ccb8" exitCode=0
Jul 28 12:14:34.925279 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.925192    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk" event={"ID":"b172ed58-35b3-4db8-b00f-17c30e6befcd","Type":"ContainerDied","Data":"d7af83a8b56c1c554694d0e99e01ffb332749ac2f0c60226a5e410a573e1ccb8"}
Jul 28 12:14:34.925279 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.925211    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"
Jul 28 12:14:34.925279 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.925242    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk" event={"ID":"b172ed58-35b3-4db8-b00f-17c30e6befcd","Type":"ContainerDied","Data":"518bd8ddecc3b27bb807839be0ab98f475f3ca21eb717056d7dd4ff83e83b754"}
Jul 28 12:14:34.930995 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.930978    2561 scope.go:117] "RemoveContainer" containerID="b153e69a0c1f2dd81808ed8d47360c16c22e4bccb3b143c312f754235d22bf01"
Jul 28 12:14:34.931289 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:14:34.931271    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b153e69a0c1f2dd81808ed8d47360c16c22e4bccb3b143c312f754235d22bf01\": container with ID starting with b153e69a0c1f2dd81808ed8d47360c16c22e4bccb3b143c312f754235d22bf01 not found: ID does not exist" containerID="b153e69a0c1f2dd81808ed8d47360c16c22e4bccb3b143c312f754235d22bf01"
Jul 28 12:14:34.931350 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.931297    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b153e69a0c1f2dd81808ed8d47360c16c22e4bccb3b143c312f754235d22bf01"} err="failed to get container status \"b153e69a0c1f2dd81808ed8d47360c16c22e4bccb3b143c312f754235d22bf01\": rpc error: code = NotFound desc = could not find container \"b153e69a0c1f2dd81808ed8d47360c16c22e4bccb3b143c312f754235d22bf01\": container with ID starting with b153e69a0c1f2dd81808ed8d47360c16c22e4bccb3b143c312f754235d22bf01 not found: ID does not exist"
Jul 28 12:14:34.931350 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.931313    2561 scope.go:117] "RemoveContainer" containerID="2a6300a69a2a9932d4a40f247d53d01ac93c5bb94fcd63f16e9057105b9336c3"
Jul 28 12:14:34.938611 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.938571    2561 scope.go:117] "RemoveContainer" containerID="2a6300a69a2a9932d4a40f247d53d01ac93c5bb94fcd63f16e9057105b9336c3"
Jul 28 12:14:34.938860 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:14:34.938839    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a6300a69a2a9932d4a40f247d53d01ac93c5bb94fcd63f16e9057105b9336c3\": container with ID starting with 2a6300a69a2a9932d4a40f247d53d01ac93c5bb94fcd63f16e9057105b9336c3 not found: ID does not exist" containerID="2a6300a69a2a9932d4a40f247d53d01ac93c5bb94fcd63f16e9057105b9336c3"
Jul 28 12:14:34.938911 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.938866    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a6300a69a2a9932d4a40f247d53d01ac93c5bb94fcd63f16e9057105b9336c3"} err="failed to get container status \"2a6300a69a2a9932d4a40f247d53d01ac93c5bb94fcd63f16e9057105b9336c3\": rpc error: code = NotFound desc = could not find container \"2a6300a69a2a9932d4a40f247d53d01ac93c5bb94fcd63f16e9057105b9336c3\": container with ID starting with 2a6300a69a2a9932d4a40f247d53d01ac93c5bb94fcd63f16e9057105b9336c3 not found: ID does not exist"
Jul 28 12:14:34.938911 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.938888    2561 scope.go:117] "RemoveContainer" containerID="d7af83a8b56c1c554694d0e99e01ffb332749ac2f0c60226a5e410a573e1ccb8"
Jul 28 12:14:34.945796 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.945780    2561 scope.go:117] "RemoveContainer" containerID="d7af83a8b56c1c554694d0e99e01ffb332749ac2f0c60226a5e410a573e1ccb8"
Jul 28 12:14:34.946025 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:14:34.946006    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7af83a8b56c1c554694d0e99e01ffb332749ac2f0c60226a5e410a573e1ccb8\": container with ID starting with d7af83a8b56c1c554694d0e99e01ffb332749ac2f0c60226a5e410a573e1ccb8 not found: ID does not exist" containerID="d7af83a8b56c1c554694d0e99e01ffb332749ac2f0c60226a5e410a573e1ccb8"
Jul 28 12:14:34.946091 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.946036    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7af83a8b56c1c554694d0e99e01ffb332749ac2f0c60226a5e410a573e1ccb8"} err="failed to get container status \"d7af83a8b56c1c554694d0e99e01ffb332749ac2f0c60226a5e410a573e1ccb8\": rpc error: code = NotFound desc = could not find container \"d7af83a8b56c1c554694d0e99e01ffb332749ac2f0c60226a5e410a573e1ccb8\": container with ID starting with d7af83a8b56c1c554694d0e99e01ffb332749ac2f0c60226a5e410a573e1ccb8 not found: ID does not exist"
Jul 28 12:14:34.949713 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.949696    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b172ed58-35b3-4db8-b00f-17c30e6befcd-maas-api-tls\") pod \"b172ed58-35b3-4db8-b00f-17c30e6befcd\" (UID: \"b172ed58-35b3-4db8-b00f-17c30e6befcd\") "
Jul 28 12:14:34.949802 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.949754    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzpln\" (UniqueName: \"kubernetes.io/projected/1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7-kube-api-access-jzpln\") pod \"1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7\" (UID: \"1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7\") "
Jul 28 12:14:34.949859 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.949829    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfw2s\" (UniqueName: \"kubernetes.io/projected/b172ed58-35b3-4db8-b00f-17c30e6befcd-kube-api-access-wfw2s\") pod \"b172ed58-35b3-4db8-b00f-17c30e6befcd\" (UID: \"b172ed58-35b3-4db8-b00f-17c30e6befcd\") "
Jul 28 12:14:34.949910 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.949869    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b172ed58-35b3-4db8-b00f-17c30e6befcd-openshift-service-ca\") pod \"b172ed58-35b3-4db8-b00f-17c30e6befcd\" (UID: \"b172ed58-35b3-4db8-b00f-17c30e6befcd\") "
Jul 28 12:14:34.949910 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.949899    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7-plugins-config-volume\") pod \"1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7\" (UID: \"1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7\") "
Jul 28 12:14:34.950239 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.950215    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b172ed58-35b3-4db8-b00f-17c30e6befcd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b172ed58-35b3-4db8-b00f-17c30e6befcd" (UID: "b172ed58-35b3-4db8-b00f-17c30e6befcd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:14:34.950316 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.950275    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7" (UID: "1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:14:34.951729 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.951709    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b172ed58-35b3-4db8-b00f-17c30e6befcd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b172ed58-35b3-4db8-b00f-17c30e6befcd" (UID: "b172ed58-35b3-4db8-b00f-17c30e6befcd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:14:34.951791 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.951727    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7-kube-api-access-jzpln" (OuterVolumeSpecName: "kube-api-access-jzpln") pod "1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7" (UID: "1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7"). InnerVolumeSpecName "kube-api-access-jzpln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:14:34.951894 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:34.951880    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b172ed58-35b3-4db8-b00f-17c30e6befcd-kube-api-access-wfw2s" (OuterVolumeSpecName: "kube-api-access-wfw2s") pod "b172ed58-35b3-4db8-b00f-17c30e6befcd" (UID: "b172ed58-35b3-4db8-b00f-17c30e6befcd"). InnerVolumeSpecName "kube-api-access-wfw2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:14:35.050843 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:35.050751    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnbj4\" (UniqueName: \"kubernetes.io/projected/1445878a-6a8d-46c1-93cb-2d4df75622c2-kube-api-access-bnbj4\") pod \"1445878a-6a8d-46c1-93cb-2d4df75622c2\" (UID: \"1445878a-6a8d-46c1-93cb-2d4df75622c2\") "
Jul 28 12:14:35.050843 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:35.050824    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1445878a-6a8d-46c1-93cb-2d4df75622c2-plugins-config-volume\") pod \"1445878a-6a8d-46c1-93cb-2d4df75622c2\" (UID: \"1445878a-6a8d-46c1-93cb-2d4df75622c2\") "
Jul 28 12:14:35.051066 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:35.051054    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b172ed58-35b3-4db8-b00f-17c30e6befcd-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:35.051132 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:35.051072    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzpln\" (UniqueName: \"kubernetes.io/projected/1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7-kube-api-access-jzpln\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:35.051132 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:35.051088    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfw2s\" (UniqueName: \"kubernetes.io/projected/b172ed58-35b3-4db8-b00f-17c30e6befcd-kube-api-access-wfw2s\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:35.051132 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:35.051102    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b172ed58-35b3-4db8-b00f-17c30e6befcd-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:35.051132 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:35.051115    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:35.051280 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:35.051252    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1445878a-6a8d-46c1-93cb-2d4df75622c2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1445878a-6a8d-46c1-93cb-2d4df75622c2" (UID: "1445878a-6a8d-46c1-93cb-2d4df75622c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:14:35.052900 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:35.052877    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1445878a-6a8d-46c1-93cb-2d4df75622c2-kube-api-access-bnbj4" (OuterVolumeSpecName: "kube-api-access-bnbj4") pod "1445878a-6a8d-46c1-93cb-2d4df75622c2" (UID: "1445878a-6a8d-46c1-93cb-2d4df75622c2"). InnerVolumeSpecName "kube-api-access-bnbj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:14:35.151948 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:35.151909    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnbj4\" (UniqueName: \"kubernetes.io/projected/1445878a-6a8d-46c1-93cb-2d4df75622c2-kube-api-access-bnbj4\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:35.151948 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:35.151942    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1445878a-6a8d-46c1-93cb-2d4df75622c2-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:14:35.249350 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:35.249292    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd"]
Jul 28 12:14:35.253109 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:35.253083    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44546f-5bcdc85884-xcwtd"]
Jul 28 12:14:35.263240 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:35.263213    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt"]
Jul 28 12:14:35.267226 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:35.267199    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44546f-56fb4d6c66-hqzgt"]
Jul 28 12:14:35.275656 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:35.275631    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"]
Jul 28 12:14:35.279147 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:35.279124    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-44546f-57d9b76dc-wf2wk"]
Jul 28 12:14:36.545626 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:36.545571    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1445878a-6a8d-46c1-93cb-2d4df75622c2" path="/var/lib/kubelet/pods/1445878a-6a8d-46c1-93cb-2d4df75622c2/volumes"
Jul 28 12:14:36.546137 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:36.546115    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7" path="/var/lib/kubelet/pods/1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7/volumes"
Jul 28 12:14:36.546560 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:36.546523    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b172ed58-35b3-4db8-b00f-17c30e6befcd" path="/var/lib/kubelet/pods/b172ed58-35b3-4db8-b00f-17c30e6befcd/volumes"
Jul 28 12:14:37.529097 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:37.529059    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:14:39.127670 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:39.127641    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44546f-5dqpt"]
Jul 28 12:14:39.129603 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:39.129568    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44546f-5dqpt"]
Jul 28 12:14:40.545956 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:40.545921    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48d08cf0-6f63-4635-a9d8-78463ddf8ba4" path="/var/lib/kubelet/pods/48d08cf0-6f63-4635-a9d8-78463ddf8ba4/volumes"
Jul 28 12:14:49.898000 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:49.897961    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"]
Jul 28 12:14:49.898412 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:49.898397    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b172ed58-35b3-4db8-b00f-17c30e6befcd" containerName="maas-api"
Jul 28 12:14:49.898459 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:49.898416    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b172ed58-35b3-4db8-b00f-17c30e6befcd" containerName="maas-api"
Jul 28 12:14:49.898459 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:49.898439    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1445878a-6a8d-46c1-93cb-2d4df75622c2" containerName="payload-pre-processing"
Jul 28 12:14:49.898459 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:49.898448    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1445878a-6a8d-46c1-93cb-2d4df75622c2" containerName="payload-pre-processing"
Jul 28 12:14:49.898552 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:49.898458    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7" containerName="payload-processing"
Jul 28 12:14:49.898552 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:49.898467    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7" containerName="payload-processing"
Jul 28 12:14:49.898552 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:49.898481    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48d08cf0-6f63-4635-a9d8-78463ddf8ba4" containerName="revoke-keys"
Jul 28 12:14:49.898552 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:49.898490    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="48d08cf0-6f63-4635-a9d8-78463ddf8ba4" containerName="revoke-keys"
Jul 28 12:14:49.898698 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:49.898555    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bf6a5c4-7b3e-428f-aeec-6d23d6299fc7" containerName="payload-processing"
Jul 28 12:14:49.898698 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:49.898567    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="48d08cf0-6f63-4635-a9d8-78463ddf8ba4" containerName="revoke-keys"
Jul 28 12:14:49.898698 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:49.898593    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1445878a-6a8d-46c1-93cb-2d4df75622c2" containerName="payload-pre-processing"
Jul 28 12:14:49.898698 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:49.898603    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b172ed58-35b3-4db8-b00f-17c30e6befcd" containerName="maas-api"
Jul 28 12:14:49.903568 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:49.903546    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"
Jul 28 12:14:49.905897 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:49.905874    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 12:14:49.905897 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:49.905889    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 12:14:49.906155 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:49.905933    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-560abc-serving-cert\""
Jul 28 12:14:49.906155 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:49.906002    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xchcw\""
Jul 28 12:14:49.910479 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:49.910461    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"]
Jul 28 12:14:50.074743 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.074708    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/befc67ce-cb11-425a-9146-2f679760f55d-openshift-service-ca\") pod \"maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm\" (UID: \"befc67ce-cb11-425a-9146-2f679760f55d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"
Jul 28 12:14:50.074930 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.074766    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tdhm\" (UniqueName: \"kubernetes.io/projected/befc67ce-cb11-425a-9146-2f679760f55d-kube-api-access-7tdhm\") pod \"maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm\" (UID: \"befc67ce-cb11-425a-9146-2f679760f55d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"
Jul 28 12:14:50.074930 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.074816    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/befc67ce-cb11-425a-9146-2f679760f55d-maas-api-tls\") pod \"maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm\" (UID: \"befc67ce-cb11-425a-9146-2f679760f55d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"
Jul 28 12:14:50.176130 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.176051    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/befc67ce-cb11-425a-9146-2f679760f55d-maas-api-tls\") pod \"maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm\" (UID: \"befc67ce-cb11-425a-9146-2f679760f55d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"
Jul 28 12:14:50.176279 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.176150    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/befc67ce-cb11-425a-9146-2f679760f55d-openshift-service-ca\") pod \"maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm\" (UID: \"befc67ce-cb11-425a-9146-2f679760f55d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"
Jul 28 12:14:50.176279 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.176176    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tdhm\" (UniqueName: \"kubernetes.io/projected/befc67ce-cb11-425a-9146-2f679760f55d-kube-api-access-7tdhm\") pod \"maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm\" (UID: \"befc67ce-cb11-425a-9146-2f679760f55d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"
Jul 28 12:14:50.176279 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:14:50.176200    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-serving-cert: secret "maas-api-e2e-api-a-560abc-serving-cert" not found
Jul 28 12:14:50.176279 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:14:50.176260    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befc67ce-cb11-425a-9146-2f679760f55d-maas-api-tls podName:befc67ce-cb11-425a-9146-2f679760f55d nodeName:}" failed. No retries permitted until 2026-07-28 12:14:50.676244823 +0000 UTC m=+2642.534900804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/befc67ce-cb11-425a-9146-2f679760f55d-maas-api-tls") pod "maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm" (UID: "befc67ce-cb11-425a-9146-2f679760f55d") : secret "maas-api-e2e-api-a-560abc-serving-cert" not found
Jul 28 12:14:50.176820 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.176796    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/befc67ce-cb11-425a-9146-2f679760f55d-openshift-service-ca\") pod \"maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm\" (UID: \"befc67ce-cb11-425a-9146-2f679760f55d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"
Jul 28 12:14:50.185044 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.185021    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tdhm\" (UniqueName: \"kubernetes.io/projected/befc67ce-cb11-425a-9146-2f679760f55d-kube-api-access-7tdhm\") pod \"maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm\" (UID: \"befc67ce-cb11-425a-9146-2f679760f55d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"
Jul 28 12:14:50.646039 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.646002    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d"]
Jul 28 12:14:50.649471 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.649445    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d"
Jul 28 12:14:50.651771 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.651750    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-560abc-dockercfg-kvx4c\""
Jul 28 12:14:50.651936 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.651903    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 12:14:50.652142 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.652128    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-560abc\""
Jul 28 12:14:50.652475 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.652459    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 12:14:50.659711 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.659691    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d"]
Jul 28 12:14:50.678965 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.678930    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/befc67ce-cb11-425a-9146-2f679760f55d-maas-api-tls\") pod \"maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm\" (UID: \"befc67ce-cb11-425a-9146-2f679760f55d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"
Jul 28 12:14:50.683070 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.683047    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/befc67ce-cb11-425a-9146-2f679760f55d-maas-api-tls\") pod \"maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm\" (UID: \"befc67ce-cb11-425a-9146-2f679760f55d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"
Jul 28 12:14:50.720102 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.720071    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b"]
Jul 28 12:14:50.724748 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.724663    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b"
Jul 28 12:14:50.730448 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.730427    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b"]
Jul 28 12:14:50.780253 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.780225    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxlzw\" (UniqueName: \"kubernetes.io/projected/72ae0b1b-8d17-45d5-ba92-f38078676ea6-kube-api-access-xxlzw\") pod \"payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d\" (UID: \"72ae0b1b-8d17-45d5-ba92-f38078676ea6\") " pod="openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d"
Jul 28 12:14:50.780402 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.780267    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ae0b1b-8d17-45d5-ba92-f38078676ea6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d\" (UID: \"72ae0b1b-8d17-45d5-ba92-f38078676ea6\") " pod="openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d"
Jul 28 12:14:50.816302 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.816274    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"
Jul 28 12:14:50.881826 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.881789    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxlzw\" (UniqueName: \"kubernetes.io/projected/72ae0b1b-8d17-45d5-ba92-f38078676ea6-kube-api-access-xxlzw\") pod \"payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d\" (UID: \"72ae0b1b-8d17-45d5-ba92-f38078676ea6\") " pod="openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d"
Jul 28 12:14:50.881994 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.881846    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1aa166-c49a-4c2b-9a23-10d468f4a3e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b\" (UID: \"4b1aa166-c49a-4c2b-9a23-10d468f4a3e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b"
Jul 28 12:14:50.881994 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.881876    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8krhc\" (UniqueName: \"kubernetes.io/projected/4b1aa166-c49a-4c2b-9a23-10d468f4a3e3-kube-api-access-8krhc\") pod \"payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b\" (UID: \"4b1aa166-c49a-4c2b-9a23-10d468f4a3e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b"
Jul 28 12:14:50.881994 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.881913    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ae0b1b-8d17-45d5-ba92-f38078676ea6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d\" (UID: \"72ae0b1b-8d17-45d5-ba92-f38078676ea6\") " pod="openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d"
Jul 28 12:14:50.882878 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.882850    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ae0b1b-8d17-45d5-ba92-f38078676ea6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d\" (UID: \"72ae0b1b-8d17-45d5-ba92-f38078676ea6\") " pod="openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d"
Jul 28 12:14:50.895523 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.895486    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxlzw\" (UniqueName: \"kubernetes.io/projected/72ae0b1b-8d17-45d5-ba92-f38078676ea6-kube-api-access-xxlzw\") pod \"payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d\" (UID: \"72ae0b1b-8d17-45d5-ba92-f38078676ea6\") " pod="openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d"
Jul 28 12:14:50.948014 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.947960    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"]
Jul 28 12:14:50.950295 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:14:50.950262    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbefc67ce_cb11_425a_9146_2f679760f55d.slice/crio-f69845a4c45544120f185f67604e75dc35c9419bbf1ed51b44917951bba3801d WatchSource:0}: Error finding container f69845a4c45544120f185f67604e75dc35c9419bbf1ed51b44917951bba3801d: Status 404 returned error can't find the container with id f69845a4c45544120f185f67604e75dc35c9419bbf1ed51b44917951bba3801d
Jul 28 12:14:50.960623 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.960598    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d"
Jul 28 12:14:50.982931 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.982904    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1aa166-c49a-4c2b-9a23-10d468f4a3e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b\" (UID: \"4b1aa166-c49a-4c2b-9a23-10d468f4a3e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b"
Jul 28 12:14:50.983043 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.982950    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8krhc\" (UniqueName: \"kubernetes.io/projected/4b1aa166-c49a-4c2b-9a23-10d468f4a3e3-kube-api-access-8krhc\") pod \"payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b\" (UID: \"4b1aa166-c49a-4c2b-9a23-10d468f4a3e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b"
Jul 28 12:14:50.983786 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.983761    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1aa166-c49a-4c2b-9a23-10d468f4a3e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b\" (UID: \"4b1aa166-c49a-4c2b-9a23-10d468f4a3e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b"
Jul 28 12:14:50.990493 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.990471    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8krhc\" (UniqueName: \"kubernetes.io/projected/4b1aa166-c49a-4c2b-9a23-10d468f4a3e3-kube-api-access-8krhc\") pod \"payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b\" (UID: \"4b1aa166-c49a-4c2b-9a23-10d468f4a3e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b"
Jul 28 12:14:50.997833 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:50.997808    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm" event={"ID":"befc67ce-cb11-425a-9146-2f679760f55d","Type":"ContainerStarted","Data":"f69845a4c45544120f185f67604e75dc35c9419bbf1ed51b44917951bba3801d"}
Jul 28 12:14:51.036784 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:51.036750    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b"
Jul 28 12:14:51.117363 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:51.117339    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d"]
Jul 28 12:14:51.119175 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:14:51.119146    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ae0b1b_8d17_45d5_ba92_f38078676ea6.slice/crio-ac2d4a1f87f6c5a6dd18054dca4f69d79975167056b379d22f0fd2b870a2b13f WatchSource:0}: Error finding container ac2d4a1f87f6c5a6dd18054dca4f69d79975167056b379d22f0fd2b870a2b13f: Status 404 returned error can't find the container with id ac2d4a1f87f6c5a6dd18054dca4f69d79975167056b379d22f0fd2b870a2b13f
Jul 28 12:14:51.176011 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:51.175973    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b"]
Jul 28 12:14:51.180758 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:14:51.178671    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1aa166_c49a_4c2b_9a23_10d468f4a3e3.slice/crio-0677684e3232c839cba5be50fa643fc6e25886401264f5be58b061b8f0833b0b WatchSource:0}: Error finding container 0677684e3232c839cba5be50fa643fc6e25886401264f5be58b061b8f0833b0b: Status 404 returned error can't find the container with id 0677684e3232c839cba5be50fa643fc6e25886401264f5be58b061b8f0833b0b
Jul 28 12:14:52.004073 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:52.004036    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d" event={"ID":"72ae0b1b-8d17-45d5-ba92-f38078676ea6","Type":"ContainerStarted","Data":"18ee013b79b3886691e8bbe17b1d482f4634bc115b5bc8ae68f83e0b38720684"}
Jul 28 12:14:52.004551 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:52.004099    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d"
Jul 28 12:14:52.004551 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:52.004117    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d" event={"ID":"72ae0b1b-8d17-45d5-ba92-f38078676ea6","Type":"ContainerStarted","Data":"ac2d4a1f87f6c5a6dd18054dca4f69d79975167056b379d22f0fd2b870a2b13f"}
Jul 28 12:14:52.005620 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:52.005596    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm" event={"ID":"befc67ce-cb11-425a-9146-2f679760f55d","Type":"ContainerStarted","Data":"e21b485d5670601187480d9431e3a65b5aaa4501463b0f23cbd9ece46fe51540"}
Jul 28 12:14:52.005745 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:52.005652    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"
Jul 28 12:14:52.007204 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:52.007176    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b" event={"ID":"4b1aa166-c49a-4c2b-9a23-10d468f4a3e3","Type":"ContainerStarted","Data":"79c65ff4dbea37a2c0ba3a08a351c7dead80f1a94b9f3e68d945b490dd9573dd"}
Jul 28 12:14:52.007282 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:52.007213    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b" event={"ID":"4b1aa166-c49a-4c2b-9a23-10d468f4a3e3","Type":"ContainerStarted","Data":"0677684e3232c839cba5be50fa643fc6e25886401264f5be58b061b8f0833b0b"}
Jul 28 12:14:52.007331 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:52.007303    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b"
Jul 28 12:14:52.021233 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:52.021192    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d" podStartSLOduration=2.021178763 podStartE2EDuration="2.021178763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:14:52.020305706 +0000 UTC m=+2643.878961707" watchObservedRunningTime="2026-07-28 12:14:52.021178763 +0000 UTC m=+2643.879834741"
Jul 28 12:14:52.039240 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:52.039205    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm" podStartSLOduration=2.638059985 podStartE2EDuration="3.039193758s" totalImagesPullingTime="401.133773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:14:52.038136168 +0000 UTC m=+2643.896792193" watchObservedRunningTime="2026-07-28 12:14:52.039193758 +0000 UTC m=+2643.897849735"
Jul 28 12:14:52.053200 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:52.053157    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b" podStartSLOduration=2.053147666 podStartE2EDuration="2.053147666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:14:52.052846607 +0000 UTC m=+2643.911502605" watchObservedRunningTime="2026-07-28 12:14:52.053147666 +0000 UTC m=+2643.911803666"
Jul 28 12:14:58.017423 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.017394    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"
Jul 28 12:14:58.190780 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.190750    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"]
Jul 28 12:14:58.194551 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.194530    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"
Jul 28 12:14:58.196526 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.196506    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-88898f-serving-cert\""
Jul 28 12:14:58.205973 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.205945    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"]
Jul 28 12:14:58.246186 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.246164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/815155b8-8e4a-423d-9dd1-1223f3c9ae9b-openshift-service-ca\") pod \"maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb\" (UID: \"815155b8-8e4a-423d-9dd1-1223f3c9ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"
Jul 28 12:14:58.246305 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.246199    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glsbq\" (UniqueName: \"kubernetes.io/projected/815155b8-8e4a-423d-9dd1-1223f3c9ae9b-kube-api-access-glsbq\") pod \"maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb\" (UID: \"815155b8-8e4a-423d-9dd1-1223f3c9ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"
Jul 28 12:14:58.246381 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.246300    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/815155b8-8e4a-423d-9dd1-1223f3c9ae9b-maas-api-tls\") pod \"maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb\" (UID: \"815155b8-8e4a-423d-9dd1-1223f3c9ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"
Jul 28 12:14:58.337893 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.337869    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:14:58.347058 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.347041    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/815155b8-8e4a-423d-9dd1-1223f3c9ae9b-maas-api-tls\") pod \"maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb\" (UID: \"815155b8-8e4a-423d-9dd1-1223f3c9ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"
Jul 28 12:14:58.347140 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.347072    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/815155b8-8e4a-423d-9dd1-1223f3c9ae9b-openshift-service-ca\") pod \"maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb\" (UID: \"815155b8-8e4a-423d-9dd1-1223f3c9ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"
Jul 28 12:14:58.347140 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.347097    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glsbq\" (UniqueName: \"kubernetes.io/projected/815155b8-8e4a-423d-9dd1-1223f3c9ae9b-kube-api-access-glsbq\") pod \"maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb\" (UID: \"815155b8-8e4a-423d-9dd1-1223f3c9ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"
Jul 28 12:14:58.347701 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.347680    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/815155b8-8e4a-423d-9dd1-1223f3c9ae9b-openshift-service-ca\") pod \"maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb\" (UID: \"815155b8-8e4a-423d-9dd1-1223f3c9ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"
Jul 28 12:14:58.349326 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.349303    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/815155b8-8e4a-423d-9dd1-1223f3c9ae9b-maas-api-tls\") pod \"maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb\" (UID: \"815155b8-8e4a-423d-9dd1-1223f3c9ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"
Jul 28 12:14:58.356067 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.356036    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glsbq\" (UniqueName: \"kubernetes.io/projected/815155b8-8e4a-423d-9dd1-1223f3c9ae9b-kube-api-access-glsbq\") pod \"maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb\" (UID: \"815155b8-8e4a-423d-9dd1-1223f3c9ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"
Jul 28 12:14:58.505299 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.505263    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"
Jul 28 12:14:58.634400 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.634373    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"]
Jul 28 12:14:58.636799 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:14:58.636773    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815155b8_8e4a_423d_9dd1_1223f3c9ae9b.slice/crio-d136370154f08c162be1b826241f5a65b8660697434758ac95a233cd29790b9c WatchSource:0}: Error finding container d136370154f08c162be1b826241f5a65b8660697434758ac95a233cd29790b9c: Status 404 returned error can't find the container with id d136370154f08c162be1b826241f5a65b8660697434758ac95a233cd29790b9c
Jul 28 12:14:58.882637 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.882597    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj"]
Jul 28 12:14:58.887184 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.887121    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj"
Jul 28 12:14:58.889225 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.889208    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-88898f\""
Jul 28 12:14:58.898317 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.898292    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj"]
Jul 28 12:14:58.953310 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.953290    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c12a757-3863-4006-85d9-0a8e534c3911-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj\" (UID: \"3c12a757-3863-4006-85d9-0a8e534c3911\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj"
Jul 28 12:14:58.953369 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:58.953335    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cjlv\" (UniqueName: \"kubernetes.io/projected/3c12a757-3863-4006-85d9-0a8e534c3911-kube-api-access-7cjlv\") pod \"payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj\" (UID: \"3c12a757-3863-4006-85d9-0a8e534c3911\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj"
Jul 28 12:14:59.038558 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:59.038527    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb" event={"ID":"815155b8-8e4a-423d-9dd1-1223f3c9ae9b","Type":"ContainerStarted","Data":"87b3a78b056cbbfc2bcb71b86adda18b51823832898524dac1b5e5b79ed98753"}
Jul 28 12:14:59.038919 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:59.038590    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb" event={"ID":"815155b8-8e4a-423d-9dd1-1223f3c9ae9b","Type":"ContainerStarted","Data":"d136370154f08c162be1b826241f5a65b8660697434758ac95a233cd29790b9c"}
Jul 28 12:14:59.038919 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:59.038680    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"
Jul 28 12:14:59.054346 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:59.054322    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c12a757-3863-4006-85d9-0a8e534c3911-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj\" (UID: \"3c12a757-3863-4006-85d9-0a8e534c3911\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj"
Jul 28 12:14:59.054403 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:59.054358    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cjlv\" (UniqueName: \"kubernetes.io/projected/3c12a757-3863-4006-85d9-0a8e534c3911-kube-api-access-7cjlv\") pod \"payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj\" (UID: \"3c12a757-3863-4006-85d9-0a8e534c3911\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj"
Jul 28 12:14:59.055017 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:59.055001    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c12a757-3863-4006-85d9-0a8e534c3911-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj\" (UID: \"3c12a757-3863-4006-85d9-0a8e534c3911\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj"
Jul 28 12:14:59.057270 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:59.057231    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb" podStartSLOduration=0.758052042 podStartE2EDuration="1.057216967s" totalImagesPullingTime="299.164925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:14:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:14:59.055098544 +0000 UTC m=+2650.913754542" watchObservedRunningTime="2026-07-28 12:14:59.057216967 +0000 UTC m=+2650.915873012"
Jul 28 12:14:59.067954 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:59.067934    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cjlv\" (UniqueName: \"kubernetes.io/projected/3c12a757-3863-4006-85d9-0a8e534c3911-kube-api-access-7cjlv\") pod \"payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj\" (UID: \"3c12a757-3863-4006-85d9-0a8e534c3911\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj"
Jul 28 12:14:59.196768 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:59.196691    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj"
Jul 28 12:14:59.325394 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:14:59.325364    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj"]
Jul 28 12:14:59.326562 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:14:59.326533    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c12a757_3863_4006_85d9_0a8e534c3911.slice/crio-05681a5b8f3e54cbc6299751d2027ddd39f9f9f62ed7bc52bbc3d53115297e81 WatchSource:0}: Error finding container 05681a5b8f3e54cbc6299751d2027ddd39f9f9f62ed7bc52bbc3d53115297e81: Status 404 returned error can't find the container with id 05681a5b8f3e54cbc6299751d2027ddd39f9f9f62ed7bc52bbc3d53115297e81
Jul 28 12:15:00.044590 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:00.044541    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj" event={"ID":"3c12a757-3863-4006-85d9-0a8e534c3911","Type":"ContainerStarted","Data":"6a7c90e434aa18a1f6c75fce88398f90d138201695139c909a7a4168449873f6"}
Jul 28 12:15:00.044988 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:00.044628    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj" event={"ID":"3c12a757-3863-4006-85d9-0a8e534c3911","Type":"ContainerStarted","Data":"05681a5b8f3e54cbc6299751d2027ddd39f9f9f62ed7bc52bbc3d53115297e81"}
Jul 28 12:15:00.044988 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:00.044764    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj"
Jul 28 12:15:00.063014 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:00.062969    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj" podStartSLOduration=2.062952295 podStartE2EDuration="2.062952295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:15:00.06225558 +0000 UTC m=+2651.920911578" watchObservedRunningTime="2026-07-28 12:15:00.062952295 +0000 UTC m=+2651.921608293"
Jul 28 12:15:03.014242 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:03.014206    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b"
Jul 28 12:15:03.014755 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:03.014363    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d"
Jul 28 12:15:05.051832 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:05.051804    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"
Jul 28 12:15:11.051097 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:11.051067    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj"
Jul 28 12:15:12.834446 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:12.834414    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:15:26.559339 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:26.559304    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"]
Jul 28 12:15:26.559971 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:26.559516    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm" podUID="befc67ce-cb11-425a-9146-2f679760f55d" containerName="maas-api" containerID="cri-o://e21b485d5670601187480d9431e3a65b5aaa4501463b0f23cbd9ece46fe51540" gracePeriod=30
Jul 28 12:15:26.631528 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:26.631494    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d"]
Jul 28 12:15:26.631804 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:26.631761    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d" podUID="72ae0b1b-8d17-45d5-ba92-f38078676ea6" containerName="payload-processing" containerID="cri-o://18ee013b79b3886691e8bbe17b1d482f4634bc115b5bc8ae68f83e0b38720684" gracePeriod=30
Jul 28 12:15:26.640498 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:26.640470    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b"]
Jul 28 12:15:26.640754 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:26.640729    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b" podUID="4b1aa166-c49a-4c2b-9a23-10d468f4a3e3" containerName="payload-pre-processing" containerID="cri-o://79c65ff4dbea37a2c0ba3a08a351c7dead80f1a94b9f3e68d945b490dd9573dd" gracePeriod=30
Jul 28 12:15:26.912203 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:26.912171    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d"
Jul 28 12:15:26.914933 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:26.914916    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"
Jul 28 12:15:26.928070 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:26.928051    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b"
Jul 28 12:15:26.988720 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:26.988690    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxlzw\" (UniqueName: \"kubernetes.io/projected/72ae0b1b-8d17-45d5-ba92-f38078676ea6-kube-api-access-xxlzw\") pod \"72ae0b1b-8d17-45d5-ba92-f38078676ea6\" (UID: \"72ae0b1b-8d17-45d5-ba92-f38078676ea6\") "
Jul 28 12:15:26.988720 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:26.988723    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/befc67ce-cb11-425a-9146-2f679760f55d-openshift-service-ca\") pod \"befc67ce-cb11-425a-9146-2f679760f55d\" (UID: \"befc67ce-cb11-425a-9146-2f679760f55d\") "
Jul 28 12:15:26.988913 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:26.988746    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/befc67ce-cb11-425a-9146-2f679760f55d-maas-api-tls\") pod \"befc67ce-cb11-425a-9146-2f679760f55d\" (UID: \"befc67ce-cb11-425a-9146-2f679760f55d\") "
Jul 28 12:15:26.988913 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:26.988818    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tdhm\" (UniqueName: \"kubernetes.io/projected/befc67ce-cb11-425a-9146-2f679760f55d-kube-api-access-7tdhm\") pod \"befc67ce-cb11-425a-9146-2f679760f55d\" (UID: \"befc67ce-cb11-425a-9146-2f679760f55d\") "
Jul 28 12:15:26.988988 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:26.988968    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ae0b1b-8d17-45d5-ba92-f38078676ea6-plugins-config-volume\") pod \"72ae0b1b-8d17-45d5-ba92-f38078676ea6\" (UID: \"72ae0b1b-8d17-45d5-ba92-f38078676ea6\") "
Jul 28 12:15:26.989035 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:26.989021    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/befc67ce-cb11-425a-9146-2f679760f55d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "befc67ce-cb11-425a-9146-2f679760f55d" (UID: "befc67ce-cb11-425a-9146-2f679760f55d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:15:26.989286 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:26.989249    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/befc67ce-cb11-425a-9146-2f679760f55d-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:15:26.989413 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:26.989307    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ae0b1b-8d17-45d5-ba92-f38078676ea6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "72ae0b1b-8d17-45d5-ba92-f38078676ea6" (UID: "72ae0b1b-8d17-45d5-ba92-f38078676ea6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:15:26.990791 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:26.990774    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/befc67ce-cb11-425a-9146-2f679760f55d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "befc67ce-cb11-425a-9146-2f679760f55d" (UID: "befc67ce-cb11-425a-9146-2f679760f55d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:15:26.991167 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:26.991148    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/befc67ce-cb11-425a-9146-2f679760f55d-kube-api-access-7tdhm" (OuterVolumeSpecName: "kube-api-access-7tdhm") pod "befc67ce-cb11-425a-9146-2f679760f55d" (UID: "befc67ce-cb11-425a-9146-2f679760f55d"). InnerVolumeSpecName "kube-api-access-7tdhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:15:26.991167 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:26.991155    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ae0b1b-8d17-45d5-ba92-f38078676ea6-kube-api-access-xxlzw" (OuterVolumeSpecName: "kube-api-access-xxlzw") pod "72ae0b1b-8d17-45d5-ba92-f38078676ea6" (UID: "72ae0b1b-8d17-45d5-ba92-f38078676ea6"). InnerVolumeSpecName "kube-api-access-xxlzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:15:27.089766 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.089698    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8krhc\" (UniqueName: \"kubernetes.io/projected/4b1aa166-c49a-4c2b-9a23-10d468f4a3e3-kube-api-access-8krhc\") pod \"4b1aa166-c49a-4c2b-9a23-10d468f4a3e3\" (UID: \"4b1aa166-c49a-4c2b-9a23-10d468f4a3e3\") "
Jul 28 12:15:27.089766 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.089763    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1aa166-c49a-4c2b-9a23-10d468f4a3e3-plugins-config-volume\") pod \"4b1aa166-c49a-4c2b-9a23-10d468f4a3e3\" (UID: \"4b1aa166-c49a-4c2b-9a23-10d468f4a3e3\") "
Jul 28 12:15:27.089908 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.089899    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tdhm\" (UniqueName: \"kubernetes.io/projected/befc67ce-cb11-425a-9146-2f679760f55d-kube-api-access-7tdhm\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:15:27.089945 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.089909    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ae0b1b-8d17-45d5-ba92-f38078676ea6-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:15:27.089945 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.089918    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxlzw\" (UniqueName: \"kubernetes.io/projected/72ae0b1b-8d17-45d5-ba92-f38078676ea6-kube-api-access-xxlzw\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:15:27.089945 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.089926    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/befc67ce-cb11-425a-9146-2f679760f55d-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:15:27.090140 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.090120    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1aa166-c49a-4c2b-9a23-10d468f4a3e3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b1aa166-c49a-4c2b-9a23-10d468f4a3e3" (UID: "4b1aa166-c49a-4c2b-9a23-10d468f4a3e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:15:27.091652 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.091629    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1aa166-c49a-4c2b-9a23-10d468f4a3e3-kube-api-access-8krhc" (OuterVolumeSpecName: "kube-api-access-8krhc") pod "4b1aa166-c49a-4c2b-9a23-10d468f4a3e3" (UID: "4b1aa166-c49a-4c2b-9a23-10d468f4a3e3"). InnerVolumeSpecName "kube-api-access-8krhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:15:27.165871 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.165820    2561 generic.go:356] "Generic (PLEG): container finished" podID="72ae0b1b-8d17-45d5-ba92-f38078676ea6" containerID="18ee013b79b3886691e8bbe17b1d482f4634bc115b5bc8ae68f83e0b38720684" exitCode=0
Jul 28 12:15:27.165871 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.165865    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d"
Jul 28 12:15:27.166053 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.165910    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d" event={"ID":"72ae0b1b-8d17-45d5-ba92-f38078676ea6","Type":"ContainerDied","Data":"18ee013b79b3886691e8bbe17b1d482f4634bc115b5bc8ae68f83e0b38720684"}
Jul 28 12:15:27.166053 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.165968    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d" event={"ID":"72ae0b1b-8d17-45d5-ba92-f38078676ea6","Type":"ContainerDied","Data":"ac2d4a1f87f6c5a6dd18054dca4f69d79975167056b379d22f0fd2b870a2b13f"}
Jul 28 12:15:27.166053 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.166000    2561 scope.go:117] "RemoveContainer" containerID="18ee013b79b3886691e8bbe17b1d482f4634bc115b5bc8ae68f83e0b38720684"
Jul 28 12:15:27.167217 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.167188    2561 generic.go:356] "Generic (PLEG): container finished" podID="befc67ce-cb11-425a-9146-2f679760f55d" containerID="e21b485d5670601187480d9431e3a65b5aaa4501463b0f23cbd9ece46fe51540" exitCode=0
Jul 28 12:15:27.167288 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.167234    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"
Jul 28 12:15:27.167288 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.167240    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm" event={"ID":"befc67ce-cb11-425a-9146-2f679760f55d","Type":"ContainerDied","Data":"e21b485d5670601187480d9431e3a65b5aaa4501463b0f23cbd9ece46fe51540"}
Jul 28 12:15:27.167399 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.167297    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm" event={"ID":"befc67ce-cb11-425a-9146-2f679760f55d","Type":"ContainerDied","Data":"f69845a4c45544120f185f67604e75dc35c9419bbf1ed51b44917951bba3801d"}
Jul 28 12:15:27.168850 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.168824    2561 generic.go:356] "Generic (PLEG): container finished" podID="4b1aa166-c49a-4c2b-9a23-10d468f4a3e3" containerID="79c65ff4dbea37a2c0ba3a08a351c7dead80f1a94b9f3e68d945b490dd9573dd" exitCode=0
Jul 28 12:15:27.168963 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.168908    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b" event={"ID":"4b1aa166-c49a-4c2b-9a23-10d468f4a3e3","Type":"ContainerDied","Data":"79c65ff4dbea37a2c0ba3a08a351c7dead80f1a94b9f3e68d945b490dd9573dd"}
Jul 28 12:15:27.168963 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.168948    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b" event={"ID":"4b1aa166-c49a-4c2b-9a23-10d468f4a3e3","Type":"ContainerDied","Data":"0677684e3232c839cba5be50fa643fc6e25886401264f5be58b061b8f0833b0b"}
Jul 28 12:15:27.169078 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.169009    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b"
Jul 28 12:15:27.175032 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.175016    2561 scope.go:117] "RemoveContainer" containerID="18ee013b79b3886691e8bbe17b1d482f4634bc115b5bc8ae68f83e0b38720684"
Jul 28 12:15:27.175293 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:15:27.175267    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ee013b79b3886691e8bbe17b1d482f4634bc115b5bc8ae68f83e0b38720684\": container with ID starting with 18ee013b79b3886691e8bbe17b1d482f4634bc115b5bc8ae68f83e0b38720684 not found: ID does not exist" containerID="18ee013b79b3886691e8bbe17b1d482f4634bc115b5bc8ae68f83e0b38720684"
Jul 28 12:15:27.175415 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.175304    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ee013b79b3886691e8bbe17b1d482f4634bc115b5bc8ae68f83e0b38720684"} err="failed to get container status \"18ee013b79b3886691e8bbe17b1d482f4634bc115b5bc8ae68f83e0b38720684\": rpc error: code = NotFound desc = could not find container \"18ee013b79b3886691e8bbe17b1d482f4634bc115b5bc8ae68f83e0b38720684\": container with ID starting with 18ee013b79b3886691e8bbe17b1d482f4634bc115b5bc8ae68f83e0b38720684 not found: ID does not exist"
Jul 28 12:15:27.175415 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.175326    2561 scope.go:117] "RemoveContainer" containerID="e21b485d5670601187480d9431e3a65b5aaa4501463b0f23cbd9ece46fe51540"
Jul 28 12:15:27.184841 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.184826    2561 scope.go:117] "RemoveContainer" containerID="e21b485d5670601187480d9431e3a65b5aaa4501463b0f23cbd9ece46fe51540"
Jul 28 12:15:27.185085 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:15:27.185066    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e21b485d5670601187480d9431e3a65b5aaa4501463b0f23cbd9ece46fe51540\": container with ID starting with e21b485d5670601187480d9431e3a65b5aaa4501463b0f23cbd9ece46fe51540 not found: ID does not exist" containerID="e21b485d5670601187480d9431e3a65b5aaa4501463b0f23cbd9ece46fe51540"
Jul 28 12:15:27.185159 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.185095    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e21b485d5670601187480d9431e3a65b5aaa4501463b0f23cbd9ece46fe51540"} err="failed to get container status \"e21b485d5670601187480d9431e3a65b5aaa4501463b0f23cbd9ece46fe51540\": rpc error: code = NotFound desc = could not find container \"e21b485d5670601187480d9431e3a65b5aaa4501463b0f23cbd9ece46fe51540\": container with ID starting with e21b485d5670601187480d9431e3a65b5aaa4501463b0f23cbd9ece46fe51540 not found: ID does not exist"
Jul 28 12:15:27.185159 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.185117    2561 scope.go:117] "RemoveContainer" containerID="79c65ff4dbea37a2c0ba3a08a351c7dead80f1a94b9f3e68d945b490dd9573dd"
Jul 28 12:15:27.190521 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.190499    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1aa166-c49a-4c2b-9a23-10d468f4a3e3-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:15:27.190521 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.190519    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8krhc\" (UniqueName: \"kubernetes.io/projected/4b1aa166-c49a-4c2b-9a23-10d468f4a3e3-kube-api-access-8krhc\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:15:27.191006 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.190984    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d"]
Jul 28 12:15:27.192427 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.192325    2561 scope.go:117] "RemoveContainer" containerID="79c65ff4dbea37a2c0ba3a08a351c7dead80f1a94b9f3e68d945b490dd9573dd"
Jul 28 12:15:27.193094 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:15:27.193076    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79c65ff4dbea37a2c0ba3a08a351c7dead80f1a94b9f3e68d945b490dd9573dd\": container with ID starting with 79c65ff4dbea37a2c0ba3a08a351c7dead80f1a94b9f3e68d945b490dd9573dd not found: ID does not exist" containerID="79c65ff4dbea37a2c0ba3a08a351c7dead80f1a94b9f3e68d945b490dd9573dd"
Jul 28 12:15:27.193176 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.193100    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79c65ff4dbea37a2c0ba3a08a351c7dead80f1a94b9f3e68d945b490dd9573dd"} err="failed to get container status \"79c65ff4dbea37a2c0ba3a08a351c7dead80f1a94b9f3e68d945b490dd9573dd\": rpc error: code = NotFound desc = could not find container \"79c65ff4dbea37a2c0ba3a08a351c7dead80f1a94b9f3e68d945b490dd9573dd\": container with ID starting with 79c65ff4dbea37a2c0ba3a08a351c7dead80f1a94b9f3e68d945b490dd9573dd not found: ID does not exist"
Jul 28 12:15:27.194507 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.194490    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-560abc-857f46b6d6-zgf2d"]
Jul 28 12:15:27.203429 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.203411    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b"]
Jul 28 12:15:27.206764 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.206745    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-560abc-5bf46d6dd4-56d6b"]
Jul 28 12:15:27.215640 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.215616    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"]
Jul 28 12:15:27.221089 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:27.221071    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-560abc-5c8cfb9875-h5kqm"]
Jul 28 12:15:28.547215 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:28.547181    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b1aa166-c49a-4c2b-9a23-10d468f4a3e3" path="/var/lib/kubelet/pods/4b1aa166-c49a-4c2b-9a23-10d468f4a3e3/volumes"
Jul 28 12:15:28.547598 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:28.547523    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72ae0b1b-8d17-45d5-ba92-f38078676ea6" path="/var/lib/kubelet/pods/72ae0b1b-8d17-45d5-ba92-f38078676ea6/volumes"
Jul 28 12:15:28.547876 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:28.547863    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="befc67ce-cb11-425a-9146-2f679760f55d" path="/var/lib/kubelet/pods/befc67ce-cb11-425a-9146-2f679760f55d/volumes"
Jul 28 12:15:29.529939 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:29.529910    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:15:53.893455 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:53.893420    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"]
Jul 28 12:15:53.893875 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:53.893645    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb" podUID="815155b8-8e4a-423d-9dd1-1223f3c9ae9b" containerName="maas-api" containerID="cri-o://87b3a78b056cbbfc2bcb71b86adda18b51823832898524dac1b5e5b79ed98753" gracePeriod=30
Jul 28 12:15:53.957664 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:53.957636    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj"]
Jul 28 12:15:53.957866 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:53.957838    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj" podUID="3c12a757-3863-4006-85d9-0a8e534c3911" containerName="payload-pre-processing" containerID="cri-o://6a7c90e434aa18a1f6c75fce88398f90d138201695139c909a7a4168449873f6" gracePeriod=30
Jul 28 12:15:54.203759 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.203732    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"
Jul 28 12:15:54.224492 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.224469    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj"
Jul 28 12:15:54.293277 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.293243    2561 generic.go:356] "Generic (PLEG): container finished" podID="3c12a757-3863-4006-85d9-0a8e534c3911" containerID="6a7c90e434aa18a1f6c75fce88398f90d138201695139c909a7a4168449873f6" exitCode=0
Jul 28 12:15:54.293468 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.293288    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj"
Jul 28 12:15:54.293468 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.293328    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj" event={"ID":"3c12a757-3863-4006-85d9-0a8e534c3911","Type":"ContainerDied","Data":"6a7c90e434aa18a1f6c75fce88398f90d138201695139c909a7a4168449873f6"}
Jul 28 12:15:54.293468 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.293384    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj" event={"ID":"3c12a757-3863-4006-85d9-0a8e534c3911","Type":"ContainerDied","Data":"05681a5b8f3e54cbc6299751d2027ddd39f9f9f62ed7bc52bbc3d53115297e81"}
Jul 28 12:15:54.293468 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.293410    2561 scope.go:117] "RemoveContainer" containerID="6a7c90e434aa18a1f6c75fce88398f90d138201695139c909a7a4168449873f6"
Jul 28 12:15:54.294690 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.294673    2561 generic.go:356] "Generic (PLEG): container finished" podID="815155b8-8e4a-423d-9dd1-1223f3c9ae9b" containerID="87b3a78b056cbbfc2bcb71b86adda18b51823832898524dac1b5e5b79ed98753" exitCode=0
Jul 28 12:15:54.294776 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.294700    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb" event={"ID":"815155b8-8e4a-423d-9dd1-1223f3c9ae9b","Type":"ContainerDied","Data":"87b3a78b056cbbfc2bcb71b86adda18b51823832898524dac1b5e5b79ed98753"}
Jul 28 12:15:54.294776 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.294716    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"
Jul 28 12:15:54.294776 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.294732    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb" event={"ID":"815155b8-8e4a-423d-9dd1-1223f3c9ae9b","Type":"ContainerDied","Data":"d136370154f08c162be1b826241f5a65b8660697434758ac95a233cd29790b9c"}
Jul 28 12:15:54.301653 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.301638    2561 scope.go:117] "RemoveContainer" containerID="6a7c90e434aa18a1f6c75fce88398f90d138201695139c909a7a4168449873f6"
Jul 28 12:15:54.301929 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:15:54.301911    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a7c90e434aa18a1f6c75fce88398f90d138201695139c909a7a4168449873f6\": container with ID starting with 6a7c90e434aa18a1f6c75fce88398f90d138201695139c909a7a4168449873f6 not found: ID does not exist" containerID="6a7c90e434aa18a1f6c75fce88398f90d138201695139c909a7a4168449873f6"
Jul 28 12:15:54.301990 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.301939    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a7c90e434aa18a1f6c75fce88398f90d138201695139c909a7a4168449873f6"} err="failed to get container status \"6a7c90e434aa18a1f6c75fce88398f90d138201695139c909a7a4168449873f6\": rpc error: code = NotFound desc = could not find container \"6a7c90e434aa18a1f6c75fce88398f90d138201695139c909a7a4168449873f6\": container with ID starting with 6a7c90e434aa18a1f6c75fce88398f90d138201695139c909a7a4168449873f6 not found: ID does not exist"
Jul 28 12:15:54.301990 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.301956    2561 scope.go:117] "RemoveContainer" containerID="87b3a78b056cbbfc2bcb71b86adda18b51823832898524dac1b5e5b79ed98753"
Jul 28 12:15:54.307068 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.307046    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glsbq\" (UniqueName: \"kubernetes.io/projected/815155b8-8e4a-423d-9dd1-1223f3c9ae9b-kube-api-access-glsbq\") pod \"815155b8-8e4a-423d-9dd1-1223f3c9ae9b\" (UID: \"815155b8-8e4a-423d-9dd1-1223f3c9ae9b\") "
Jul 28 12:15:54.307165 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.307105    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cjlv\" (UniqueName: \"kubernetes.io/projected/3c12a757-3863-4006-85d9-0a8e534c3911-kube-api-access-7cjlv\") pod \"3c12a757-3863-4006-85d9-0a8e534c3911\" (UID: \"3c12a757-3863-4006-85d9-0a8e534c3911\") "
Jul 28 12:15:54.307165 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.307154    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/815155b8-8e4a-423d-9dd1-1223f3c9ae9b-openshift-service-ca\") pod \"815155b8-8e4a-423d-9dd1-1223f3c9ae9b\" (UID: \"815155b8-8e4a-423d-9dd1-1223f3c9ae9b\") "
Jul 28 12:15:54.307285 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.307182    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/815155b8-8e4a-423d-9dd1-1223f3c9ae9b-maas-api-tls\") pod \"815155b8-8e4a-423d-9dd1-1223f3c9ae9b\" (UID: \"815155b8-8e4a-423d-9dd1-1223f3c9ae9b\") "
Jul 28 12:15:54.307285 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.307213    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c12a757-3863-4006-85d9-0a8e534c3911-plugins-config-volume\") pod \"3c12a757-3863-4006-85d9-0a8e534c3911\" (UID: \"3c12a757-3863-4006-85d9-0a8e534c3911\") "
Jul 28 12:15:54.307573 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.307503    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/815155b8-8e4a-423d-9dd1-1223f3c9ae9b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "815155b8-8e4a-423d-9dd1-1223f3c9ae9b" (UID: "815155b8-8e4a-423d-9dd1-1223f3c9ae9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:15:54.307748 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.307723    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c12a757-3863-4006-85d9-0a8e534c3911-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3c12a757-3863-4006-85d9-0a8e534c3911" (UID: "3c12a757-3863-4006-85d9-0a8e534c3911"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:15:54.309172 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.309137    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815155b8-8e4a-423d-9dd1-1223f3c9ae9b-kube-api-access-glsbq" (OuterVolumeSpecName: "kube-api-access-glsbq") pod "815155b8-8e4a-423d-9dd1-1223f3c9ae9b" (UID: "815155b8-8e4a-423d-9dd1-1223f3c9ae9b"). InnerVolumeSpecName "kube-api-access-glsbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:15:54.309172 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.309149    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c12a757-3863-4006-85d9-0a8e534c3911-kube-api-access-7cjlv" (OuterVolumeSpecName: "kube-api-access-7cjlv") pod "3c12a757-3863-4006-85d9-0a8e534c3911" (UID: "3c12a757-3863-4006-85d9-0a8e534c3911"). InnerVolumeSpecName "kube-api-access-7cjlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:15:54.309485 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.309470    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/815155b8-8e4a-423d-9dd1-1223f3c9ae9b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "815155b8-8e4a-423d-9dd1-1223f3c9ae9b" (UID: "815155b8-8e4a-423d-9dd1-1223f3c9ae9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:15:54.309832 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.309818    2561 scope.go:117] "RemoveContainer" containerID="87b3a78b056cbbfc2bcb71b86adda18b51823832898524dac1b5e5b79ed98753"
Jul 28 12:15:54.310050 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:15:54.310032    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b3a78b056cbbfc2bcb71b86adda18b51823832898524dac1b5e5b79ed98753\": container with ID starting with 87b3a78b056cbbfc2bcb71b86adda18b51823832898524dac1b5e5b79ed98753 not found: ID does not exist" containerID="87b3a78b056cbbfc2bcb71b86adda18b51823832898524dac1b5e5b79ed98753"
Jul 28 12:15:54.310101 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.310056    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b3a78b056cbbfc2bcb71b86adda18b51823832898524dac1b5e5b79ed98753"} err="failed to get container status \"87b3a78b056cbbfc2bcb71b86adda18b51823832898524dac1b5e5b79ed98753\": rpc error: code = NotFound desc = could not find container \"87b3a78b056cbbfc2bcb71b86adda18b51823832898524dac1b5e5b79ed98753\": container with ID starting with 87b3a78b056cbbfc2bcb71b86adda18b51823832898524dac1b5e5b79ed98753 not found: ID does not exist"
Jul 28 12:15:54.408336 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.408270    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/815155b8-8e4a-423d-9dd1-1223f3c9ae9b-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:15:54.408336 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.408294    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/815155b8-8e4a-423d-9dd1-1223f3c9ae9b-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:15:54.408336 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.408305    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c12a757-3863-4006-85d9-0a8e534c3911-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:15:54.408336 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.408316    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glsbq\" (UniqueName: \"kubernetes.io/projected/815155b8-8e4a-423d-9dd1-1223f3c9ae9b-kube-api-access-glsbq\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:15:54.408336 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.408325    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cjlv\" (UniqueName: \"kubernetes.io/projected/3c12a757-3863-4006-85d9-0a8e534c3911-kube-api-access-7cjlv\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:15:54.609552 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.609524    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj"]
Jul 28 12:15:54.614795 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.614757    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-88898f-9b54fd8dd-jvjfj"]
Jul 28 12:15:54.623512 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.623490    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"]
Jul 28 12:15:54.629109 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:54.629090    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-88898f-7d9b456bf4-hzgqb"]
Jul 28 12:15:56.545613 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:56.545562    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c12a757-3863-4006-85d9-0a8e534c3911" path="/var/lib/kubelet/pods/3c12a757-3863-4006-85d9-0a8e534c3911/volumes"
Jul 28 12:15:56.546000 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:56.545941    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="815155b8-8e4a-423d-9dd1-1223f3c9ae9b" path="/var/lib/kubelet/pods/815155b8-8e4a-423d-9dd1-1223f3c9ae9b/volumes"
Jul 28 12:15:58.428984 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:15:58.428945    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:16:09.366424 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.366390    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"]
Jul 28 12:16:09.366905 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.366864    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="815155b8-8e4a-423d-9dd1-1223f3c9ae9b" containerName="maas-api"
Jul 28 12:16:09.366905 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.366884    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="815155b8-8e4a-423d-9dd1-1223f3c9ae9b" containerName="maas-api"
Jul 28 12:16:09.366905 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.366900    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b1aa166-c49a-4c2b-9a23-10d468f4a3e3" containerName="payload-pre-processing"
Jul 28 12:16:09.367077 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.366908    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b1aa166-c49a-4c2b-9a23-10d468f4a3e3" containerName="payload-pre-processing"
Jul 28 12:16:09.367077 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.366918    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="befc67ce-cb11-425a-9146-2f679760f55d" containerName="maas-api"
Jul 28 12:16:09.367077 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.366926    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="befc67ce-cb11-425a-9146-2f679760f55d" containerName="maas-api"
Jul 28 12:16:09.367077 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.366945    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c12a757-3863-4006-85d9-0a8e534c3911" containerName="payload-pre-processing"
Jul 28 12:16:09.367077 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.366953    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c12a757-3863-4006-85d9-0a8e534c3911" containerName="payload-pre-processing"
Jul 28 12:16:09.367077 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.366967    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ae0b1b-8d17-45d5-ba92-f38078676ea6" containerName="payload-processing"
Jul 28 12:16:09.367077 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.366976    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ae0b1b-8d17-45d5-ba92-f38078676ea6" containerName="payload-processing"
Jul 28 12:16:09.367077 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.367055    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="befc67ce-cb11-425a-9146-2f679760f55d" containerName="maas-api"
Jul 28 12:16:09.367077 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.367065    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b1aa166-c49a-4c2b-9a23-10d468f4a3e3" containerName="payload-pre-processing"
Jul 28 12:16:09.367077 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.367078    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="815155b8-8e4a-423d-9dd1-1223f3c9ae9b" containerName="maas-api"
Jul 28 12:16:09.367547 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.367086    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ae0b1b-8d17-45d5-ba92-f38078676ea6" containerName="payload-processing"
Jul 28 12:16:09.367547 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.367095    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c12a757-3863-4006-85d9-0a8e534c3911" containerName="payload-pre-processing"
Jul 28 12:16:09.370294 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.370272    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"
Jul 28 12:16:09.372674 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.372653    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-4d5cfe-serving-cert\""
Jul 28 12:16:09.372924 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.372683    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 12:16:09.373199 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.373177    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xchcw\""
Jul 28 12:16:09.373305 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.373232    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 12:16:09.379671 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.379648    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"]
Jul 28 12:16:09.449616 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.449566    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/830555b4-08e7-4df2-bc3f-2bbc3141c76e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx\" (UID: \"830555b4-08e7-4df2-bc3f-2bbc3141c76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"
Jul 28 12:16:09.449737 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.449678    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqr47\" (UniqueName: \"kubernetes.io/projected/830555b4-08e7-4df2-bc3f-2bbc3141c76e-kube-api-access-zqr47\") pod \"maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx\" (UID: \"830555b4-08e7-4df2-bc3f-2bbc3141c76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"
Jul 28 12:16:09.449787 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.449749    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/830555b4-08e7-4df2-bc3f-2bbc3141c76e-maas-api-tls\") pod \"maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx\" (UID: \"830555b4-08e7-4df2-bc3f-2bbc3141c76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"
Jul 28 12:16:09.550635 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.550611    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/830555b4-08e7-4df2-bc3f-2bbc3141c76e-maas-api-tls\") pod \"maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx\" (UID: \"830555b4-08e7-4df2-bc3f-2bbc3141c76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"
Jul 28 12:16:09.550768 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.550647    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/830555b4-08e7-4df2-bc3f-2bbc3141c76e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx\" (UID: \"830555b4-08e7-4df2-bc3f-2bbc3141c76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"
Jul 28 12:16:09.550768 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.550704    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqr47\" (UniqueName: \"kubernetes.io/projected/830555b4-08e7-4df2-bc3f-2bbc3141c76e-kube-api-access-zqr47\") pod \"maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx\" (UID: \"830555b4-08e7-4df2-bc3f-2bbc3141c76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"
Jul 28 12:16:09.550847 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:16:09.550786    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-serving-cert: secret "maas-api-e2e-shared-a-4d5cfe-serving-cert" not found
Jul 28 12:16:09.550847 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:16:09.550844    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/830555b4-08e7-4df2-bc3f-2bbc3141c76e-maas-api-tls podName:830555b4-08e7-4df2-bc3f-2bbc3141c76e nodeName:}" failed. No retries permitted until 2026-07-28 12:16:10.050828948 +0000 UTC m=+2721.909484925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/830555b4-08e7-4df2-bc3f-2bbc3141c76e-maas-api-tls") pod "maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx" (UID: "830555b4-08e7-4df2-bc3f-2bbc3141c76e") : secret "maas-api-e2e-shared-a-4d5cfe-serving-cert" not found
Jul 28 12:16:09.551329 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.551311    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/830555b4-08e7-4df2-bc3f-2bbc3141c76e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx\" (UID: \"830555b4-08e7-4df2-bc3f-2bbc3141c76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"
Jul 28 12:16:09.566299 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:09.566271    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqr47\" (UniqueName: \"kubernetes.io/projected/830555b4-08e7-4df2-bc3f-2bbc3141c76e-kube-api-access-zqr47\") pod \"maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx\" (UID: \"830555b4-08e7-4df2-bc3f-2bbc3141c76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"
Jul 28 12:16:10.054473 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.054439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/830555b4-08e7-4df2-bc3f-2bbc3141c76e-maas-api-tls\") pod \"maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx\" (UID: \"830555b4-08e7-4df2-bc3f-2bbc3141c76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"
Jul 28 12:16:10.056744 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.056724    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/830555b4-08e7-4df2-bc3f-2bbc3141c76e-maas-api-tls\") pod \"maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx\" (UID: \"830555b4-08e7-4df2-bc3f-2bbc3141c76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"
Jul 28 12:16:10.128135 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.128102    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54"]
Jul 28 12:16:10.131326 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.131310    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54"
Jul 28 12:16:10.133670 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.133642    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-4d5cfe\""
Jul 28 12:16:10.133793 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.133684    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-4d5cfe-dockercfg-jjp6z\""
Jul 28 12:16:10.133969 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.133951    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 12:16:10.134480 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.134462    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 12:16:10.142557 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.142535    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54"]
Jul 28 12:16:10.176240 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.176215    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6"]
Jul 28 12:16:10.179304 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.179288    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6"
Jul 28 12:16:10.188602 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.188555    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6"]
Jul 28 12:16:10.256364 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.256328    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35cb4004-f092-4b19-ae7c-5b582bb7f555-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54\" (UID: \"35cb4004-f092-4b19-ae7c-5b582bb7f555\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54"
Jul 28 12:16:10.256364 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.256367    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58592270-db06-499b-ab98-119e6331b69a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6\" (UID: \"58592270-db06-499b-ab98-119e6331b69a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6"
Jul 28 12:16:10.256602 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.256394    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgmzv\" (UniqueName: \"kubernetes.io/projected/58592270-db06-499b-ab98-119e6331b69a-kube-api-access-bgmzv\") pod \"payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6\" (UID: \"58592270-db06-499b-ab98-119e6331b69a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6"
Jul 28 12:16:10.256602 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.256451    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f5jf\" (UniqueName: \"kubernetes.io/projected/35cb4004-f092-4b19-ae7c-5b582bb7f555-kube-api-access-7f5jf\") pod \"payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54\" (UID: \"35cb4004-f092-4b19-ae7c-5b582bb7f555\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54"
Jul 28 12:16:10.281281 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.281245    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"
Jul 28 12:16:10.362757 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.360269    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35cb4004-f092-4b19-ae7c-5b582bb7f555-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54\" (UID: \"35cb4004-f092-4b19-ae7c-5b582bb7f555\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54"
Jul 28 12:16:10.362757 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.360329    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58592270-db06-499b-ab98-119e6331b69a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6\" (UID: \"58592270-db06-499b-ab98-119e6331b69a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6"
Jul 28 12:16:10.362757 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.360361    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgmzv\" (UniqueName: \"kubernetes.io/projected/58592270-db06-499b-ab98-119e6331b69a-kube-api-access-bgmzv\") pod \"payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6\" (UID: \"58592270-db06-499b-ab98-119e6331b69a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6"
Jul 28 12:16:10.362757 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.360422    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f5jf\" (UniqueName: \"kubernetes.io/projected/35cb4004-f092-4b19-ae7c-5b582bb7f555-kube-api-access-7f5jf\") pod \"payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54\" (UID: \"35cb4004-f092-4b19-ae7c-5b582bb7f555\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54"
Jul 28 12:16:10.362757 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.361488    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35cb4004-f092-4b19-ae7c-5b582bb7f555-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54\" (UID: \"35cb4004-f092-4b19-ae7c-5b582bb7f555\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54"
Jul 28 12:16:10.362757 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.362271    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58592270-db06-499b-ab98-119e6331b69a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6\" (UID: \"58592270-db06-499b-ab98-119e6331b69a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6"
Jul 28 12:16:10.371805 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.371778    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgmzv\" (UniqueName: \"kubernetes.io/projected/58592270-db06-499b-ab98-119e6331b69a-kube-api-access-bgmzv\") pod \"payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6\" (UID: \"58592270-db06-499b-ab98-119e6331b69a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6"
Jul 28 12:16:10.372133 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.371976    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f5jf\" (UniqueName: \"kubernetes.io/projected/35cb4004-f092-4b19-ae7c-5b582bb7f555-kube-api-access-7f5jf\") pod \"payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54\" (UID: \"35cb4004-f092-4b19-ae7c-5b582bb7f555\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54"
Jul 28 12:16:10.431033 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.431000    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"]
Jul 28 12:16:10.433957 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:16:10.433936    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830555b4_08e7_4df2_bc3f_2bbc3141c76e.slice/crio-b59798d3cc480f813b57b11d43b61b84f2792d19f0504e30423402e48d16b02a WatchSource:0}: Error finding container b59798d3cc480f813b57b11d43b61b84f2792d19f0504e30423402e48d16b02a: Status 404 returned error can't find the container with id b59798d3cc480f813b57b11d43b61b84f2792d19f0504e30423402e48d16b02a
Jul 28 12:16:10.441906 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.441885    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54"
Jul 28 12:16:10.489442 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.489408    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6"
Jul 28 12:16:10.584573 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.584177    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54"]
Jul 28 12:16:10.591670 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:16:10.591635    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cb4004_f092_4b19_ae7c_5b582bb7f555.slice/crio-2a6603b347c919fce041448907fa4af93884c16f0ebe672d9cec6bc57f7772cd WatchSource:0}: Error finding container 2a6603b347c919fce041448907fa4af93884c16f0ebe672d9cec6bc57f7772cd: Status 404 returned error can't find the container with id 2a6603b347c919fce041448907fa4af93884c16f0ebe672d9cec6bc57f7772cd
Jul 28 12:16:10.634860 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:10.634838    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6"]
Jul 28 12:16:10.637182 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:16:10.637153    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58592270_db06_499b_ab98_119e6331b69a.slice/crio-b6175e05cde40215cba0894c15723adbea5d2c7351249f1b4659626bbd5a0706 WatchSource:0}: Error finding container b6175e05cde40215cba0894c15723adbea5d2c7351249f1b4659626bbd5a0706: Status 404 returned error can't find the container with id b6175e05cde40215cba0894c15723adbea5d2c7351249f1b4659626bbd5a0706
Jul 28 12:16:11.376164 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:11.376118    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx" event={"ID":"830555b4-08e7-4df2-bc3f-2bbc3141c76e","Type":"ContainerStarted","Data":"f3cc8f81402bc7aa3e46b2ab27c91dcdc19b201a9848758a936cd68beaabc921"}
Jul 28 12:16:11.376647 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:11.376185    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"
Jul 28 12:16:11.376647 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:11.376211    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx" event={"ID":"830555b4-08e7-4df2-bc3f-2bbc3141c76e","Type":"ContainerStarted","Data":"b59798d3cc480f813b57b11d43b61b84f2792d19f0504e30423402e48d16b02a"}
Jul 28 12:16:11.377999 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:11.377974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6" event={"ID":"58592270-db06-499b-ab98-119e6331b69a","Type":"ContainerStarted","Data":"60999e73ba0b3a29abacb3472dc38cbc91f05b79c370d77e62670aa245c4fdb4"}
Jul 28 12:16:11.378132 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:11.378020    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6" event={"ID":"58592270-db06-499b-ab98-119e6331b69a","Type":"ContainerStarted","Data":"b6175e05cde40215cba0894c15723adbea5d2c7351249f1b4659626bbd5a0706"}
Jul 28 12:16:11.378132 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:11.378050    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6"
Jul 28 12:16:11.379504 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:11.379480    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54" event={"ID":"35cb4004-f092-4b19-ae7c-5b582bb7f555","Type":"ContainerStarted","Data":"b8083a2b6f4669b9c0cb8a58d26c92b8db2e428201acf1676491e526b1ffc461"}
Jul 28 12:16:11.379601 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:11.379527    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54" event={"ID":"35cb4004-f092-4b19-ae7c-5b582bb7f555","Type":"ContainerStarted","Data":"2a6603b347c919fce041448907fa4af93884c16f0ebe672d9cec6bc57f7772cd"}
Jul 28 12:16:11.379601 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:11.379565    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54"
Jul 28 12:16:11.391729 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:11.391688    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx" podStartSLOduration=2.042327631 podStartE2EDuration="2.391666101s" totalImagesPullingTime="349.33847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:16:11.3915667 +0000 UTC m=+2723.250222698" watchObservedRunningTime="2026-07-28 12:16:11.391666101 +0000 UTC m=+2723.250322082"
Jul 28 12:16:11.408236 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:11.408200    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54" podStartSLOduration=1.4081879 podStartE2EDuration="1.4081879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:16:11.406522385 +0000 UTC m=+2723.265178392" watchObservedRunningTime="2026-07-28 12:16:11.4081879 +0000 UTC m=+2723.266843898"
Jul 28 12:16:11.425356 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:11.425316    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6" podStartSLOduration=1.425304964 podStartE2EDuration="1.425304964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:16:11.424060924 +0000 UTC m=+2723.282716924" watchObservedRunningTime="2026-07-28 12:16:11.425304964 +0000 UTC m=+2723.283960963"
Jul 28 12:16:16.530690 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:16.530655    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:16:17.391223 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:17.391190    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"
Jul 28 12:16:17.612246 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:17.612209    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"]
Jul 28 12:16:17.616381 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:17.616356    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"
Jul 28 12:16:17.621845 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:17.621828    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-e7af0c-serving-cert\""
Jul 28 12:16:17.629484 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:17.629462    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"]
Jul 28 12:16:17.726393 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:17.726306    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/667a5e55-9c49-4cb7-80e2-7306c9382006-maas-api-tls\") pod \"maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq\" (UID: \"667a5e55-9c49-4cb7-80e2-7306c9382006\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"
Jul 28 12:16:17.726393 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:17.726373    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/667a5e55-9c49-4cb7-80e2-7306c9382006-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq\" (UID: \"667a5e55-9c49-4cb7-80e2-7306c9382006\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"
Jul 28 12:16:17.726601 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:17.726456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxdgs\" (UniqueName: \"kubernetes.io/projected/667a5e55-9c49-4cb7-80e2-7306c9382006-kube-api-access-qxdgs\") pod \"maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq\" (UID: \"667a5e55-9c49-4cb7-80e2-7306c9382006\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"
Jul 28 12:16:17.827159 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:17.827115    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxdgs\" (UniqueName: \"kubernetes.io/projected/667a5e55-9c49-4cb7-80e2-7306c9382006-kube-api-access-qxdgs\") pod \"maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq\" (UID: \"667a5e55-9c49-4cb7-80e2-7306c9382006\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"
Jul 28 12:16:17.827159 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:17.827155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/667a5e55-9c49-4cb7-80e2-7306c9382006-maas-api-tls\") pod \"maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq\" (UID: \"667a5e55-9c49-4cb7-80e2-7306c9382006\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"
Jul 28 12:16:17.827364 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:17.827204    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/667a5e55-9c49-4cb7-80e2-7306c9382006-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq\" (UID: \"667a5e55-9c49-4cb7-80e2-7306c9382006\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"
Jul 28 12:16:17.827364 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:16:17.827318    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-serving-cert: secret "maas-api-e2e-shared-b-e7af0c-serving-cert" not found
Jul 28 12:16:17.827445 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:16:17.827393    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/667a5e55-9c49-4cb7-80e2-7306c9382006-maas-api-tls podName:667a5e55-9c49-4cb7-80e2-7306c9382006 nodeName:}" failed. No retries permitted until 2026-07-28 12:16:18.327375978 +0000 UTC m=+2730.186031956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/667a5e55-9c49-4cb7-80e2-7306c9382006-maas-api-tls") pod "maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq" (UID: "667a5e55-9c49-4cb7-80e2-7306c9382006") : secret "maas-api-e2e-shared-b-e7af0c-serving-cert" not found
Jul 28 12:16:17.827797 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:17.827781    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/667a5e55-9c49-4cb7-80e2-7306c9382006-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq\" (UID: \"667a5e55-9c49-4cb7-80e2-7306c9382006\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"
Jul 28 12:16:17.838132 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:17.838112    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxdgs\" (UniqueName: \"kubernetes.io/projected/667a5e55-9c49-4cb7-80e2-7306c9382006-kube-api-access-qxdgs\") pod \"maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq\" (UID: \"667a5e55-9c49-4cb7-80e2-7306c9382006\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"
Jul 28 12:16:18.331165 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.331130    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/667a5e55-9c49-4cb7-80e2-7306c9382006-maas-api-tls\") pod \"maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq\" (UID: \"667a5e55-9c49-4cb7-80e2-7306c9382006\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"
Jul 28 12:16:18.333426 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.333404    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/667a5e55-9c49-4cb7-80e2-7306c9382006-maas-api-tls\") pod \"maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq\" (UID: \"667a5e55-9c49-4cb7-80e2-7306c9382006\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"
Jul 28 12:16:18.397049 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.397019    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk"]
Jul 28 12:16:18.400012 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.399998    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk"
Jul 28 12:16:18.402022 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.402000    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-e7af0c-dockercfg-d5wr5\""
Jul 28 12:16:18.402130 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.402008    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-e7af0c\""
Jul 28 12:16:18.411434 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.411413    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk"]
Jul 28 12:16:18.520868 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.520841    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz"]
Jul 28 12:16:18.524268 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.524253    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz"
Jul 28 12:16:18.527814 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.527766    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"
Jul 28 12:16:18.532447 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.532422    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz"]
Jul 28 12:16:18.532793 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.532768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8fba81-cbee-4737-9f72-0fed7faa029e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk\" (UID: \"fc8fba81-cbee-4737-9f72-0fed7faa029e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk"
Jul 28 12:16:18.532905 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.532815    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw9kb\" (UniqueName: \"kubernetes.io/projected/fc8fba81-cbee-4737-9f72-0fed7faa029e-kube-api-access-kw9kb\") pod \"payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk\" (UID: \"fc8fba81-cbee-4737-9f72-0fed7faa029e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk"
Jul 28 12:16:18.634633 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.634559    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25bef550-03be-4795-894a-2dca4589dac8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz\" (UID: \"25bef550-03be-4795-894a-2dca4589dac8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz"
Jul 28 12:16:18.635069 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.634713    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8fba81-cbee-4737-9f72-0fed7faa029e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk\" (UID: \"fc8fba81-cbee-4737-9f72-0fed7faa029e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk"
Jul 28 12:16:18.635069 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.634758    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw9kb\" (UniqueName: \"kubernetes.io/projected/fc8fba81-cbee-4737-9f72-0fed7faa029e-kube-api-access-kw9kb\") pod \"payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk\" (UID: \"fc8fba81-cbee-4737-9f72-0fed7faa029e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk"
Jul 28 12:16:18.635069 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.634799    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhw4g\" (UniqueName: \"kubernetes.io/projected/25bef550-03be-4795-894a-2dca4589dac8-kube-api-access-fhw4g\") pod \"payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz\" (UID: \"25bef550-03be-4795-894a-2dca4589dac8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz"
Jul 28 12:16:18.636090 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.636063    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8fba81-cbee-4737-9f72-0fed7faa029e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk\" (UID: \"fc8fba81-cbee-4737-9f72-0fed7faa029e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk"
Jul 28 12:16:18.646634 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.646606    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw9kb\" (UniqueName: \"kubernetes.io/projected/fc8fba81-cbee-4737-9f72-0fed7faa029e-kube-api-access-kw9kb\") pod \"payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk\" (UID: \"fc8fba81-cbee-4737-9f72-0fed7faa029e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk"
Jul 28 12:16:18.710746 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.710717    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk"
Jul 28 12:16:18.735641 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.735616    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25bef550-03be-4795-894a-2dca4589dac8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz\" (UID: \"25bef550-03be-4795-894a-2dca4589dac8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz"
Jul 28 12:16:18.735756 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.735701    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhw4g\" (UniqueName: \"kubernetes.io/projected/25bef550-03be-4795-894a-2dca4589dac8-kube-api-access-fhw4g\") pod \"payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz\" (UID: \"25bef550-03be-4795-894a-2dca4589dac8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz"
Jul 28 12:16:18.736289 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.736267    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25bef550-03be-4795-894a-2dca4589dac8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz\" (UID: \"25bef550-03be-4795-894a-2dca4589dac8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz"
Jul 28 12:16:18.747843 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.747815    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhw4g\" (UniqueName: \"kubernetes.io/projected/25bef550-03be-4795-894a-2dca4589dac8-kube-api-access-fhw4g\") pod \"payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz\" (UID: \"25bef550-03be-4795-894a-2dca4589dac8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz"
Jul 28 12:16:18.834977 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.834895    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz"
Jul 28 12:16:18.842693 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.842672    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk"]
Jul 28 12:16:18.845996 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:16:18.845970    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8fba81_cbee_4737_9f72_0fed7faa029e.slice/crio-43d06f4699e207d76d5931f54d9038da842e2beb8a578b83c693ef8ff6765720 WatchSource:0}: Error finding container 43d06f4699e207d76d5931f54d9038da842e2beb8a578b83c693ef8ff6765720: Status 404 returned error can't find the container with id 43d06f4699e207d76d5931f54d9038da842e2beb8a578b83c693ef8ff6765720
Jul 28 12:16:18.875312 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.875286    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"]
Jul 28 12:16:18.882770 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:16:18.882731    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667a5e55_9c49_4cb7_80e2_7306c9382006.slice/crio-78dc76b74dca63a80c3a33f98e5a236c10b44bba5b5eddf1e88d4cdd2b1b0132 WatchSource:0}: Error finding container 78dc76b74dca63a80c3a33f98e5a236c10b44bba5b5eddf1e88d4cdd2b1b0132: Status 404 returned error can't find the container with id 78dc76b74dca63a80c3a33f98e5a236c10b44bba5b5eddf1e88d4cdd2b1b0132
Jul 28 12:16:18.980953 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:18.980885    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz"]
Jul 28 12:16:18.982588 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:16:18.982543    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25bef550_03be_4795_894a_2dca4589dac8.slice/crio-6477b952f578a5fa5bb9456a56f476a416b49c20da960815740b14828809ca74 WatchSource:0}: Error finding container 6477b952f578a5fa5bb9456a56f476a416b49c20da960815740b14828809ca74: Status 404 returned error can't find the container with id 6477b952f578a5fa5bb9456a56f476a416b49c20da960815740b14828809ca74
Jul 28 12:16:19.424658 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:19.424618    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz" event={"ID":"25bef550-03be-4795-894a-2dca4589dac8","Type":"ContainerStarted","Data":"6106c851c6e2a46c4b3f725b3c59464b48d4c47099b4675c366bb09d93e4e921"}
Jul 28 12:16:19.424849 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:19.424693    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz" event={"ID":"25bef550-03be-4795-894a-2dca4589dac8","Type":"ContainerStarted","Data":"6477b952f578a5fa5bb9456a56f476a416b49c20da960815740b14828809ca74"}
Jul 28 12:16:19.424930 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:19.424860    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz"
Jul 28 12:16:19.426277 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:19.426254    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk" event={"ID":"fc8fba81-cbee-4737-9f72-0fed7faa029e","Type":"ContainerStarted","Data":"3a33ae8acf304a0106f36acb24c92e54ba96ea0148438091b3bec94ca69adcb2"}
Jul 28 12:16:19.426419 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:19.426292    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk" event={"ID":"fc8fba81-cbee-4737-9f72-0fed7faa029e","Type":"ContainerStarted","Data":"43d06f4699e207d76d5931f54d9038da842e2beb8a578b83c693ef8ff6765720"}
Jul 28 12:16:19.426419 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:19.426321    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk"
Jul 28 12:16:19.427843 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:19.427817    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq" event={"ID":"667a5e55-9c49-4cb7-80e2-7306c9382006","Type":"ContainerStarted","Data":"8d598b641fc0289486bc901b1a7596a5c032fb2e11a8decd0d699d68c2e87c35"}
Jul 28 12:16:19.427954 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:19.427861    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq" event={"ID":"667a5e55-9c49-4cb7-80e2-7306c9382006","Type":"ContainerStarted","Data":"78dc76b74dca63a80c3a33f98e5a236c10b44bba5b5eddf1e88d4cdd2b1b0132"}
Jul 28 12:16:19.427954 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:19.427901    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"
Jul 28 12:16:19.442036 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:19.441997    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz" podStartSLOduration=1.441986382 podStartE2EDuration="1.441986382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:16:19.440206857 +0000 UTC m=+2731.298862859" watchObservedRunningTime="2026-07-28 12:16:19.441986382 +0000 UTC m=+2731.300642382"
Jul 28 12:16:19.458188 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:19.458086    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq" podStartSLOduration=2.137512569 podStartE2EDuration="2.458068592s" totalImagesPullingTime="320.556023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:16:19.456421921 +0000 UTC m=+2731.315077922" watchObservedRunningTime="2026-07-28 12:16:19.458068592 +0000 UTC m=+2731.316724570"
Jul 28 12:16:19.473847 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:19.473803    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk" podStartSLOduration=1.4737912770000001 podStartE2EDuration="1.473791277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:16:19.472413935 +0000 UTC m=+2731.331069971" watchObservedRunningTime="2026-07-28 12:16:19.473791277 +0000 UTC m=+2731.332447326"
Jul 28 12:16:22.387091 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:22.387061    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54"
Jul 28 12:16:22.387518 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:22.387127    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6"
Jul 28 12:16:25.438844 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:25.438816    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"
Jul 28 12:16:28.533748 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:28.533713    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:16:30.435383 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:30.435353    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk"
Jul 28 12:16:30.435979 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:30.435652    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz"
Jul 28 12:16:42.807735 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.807704    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"]
Jul 28 12:16:42.811247 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.811228    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:42.814570 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.814544    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4d5cfe\"/\"default-dockercfg-8gj6v\""
Jul 28 12:16:42.814736 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.814546    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4d5cfe\"/\"auth-test-model-4d5cfe-kserve-self-signed-certs\""
Jul 28 12:16:42.814817 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.814610    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4d5cfe\"/\"openshift-service-ca.crt\""
Jul 28 12:16:42.814817 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.814800    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4d5cfe\"/\"kube-root-ca.crt\""
Jul 28 12:16:42.828092 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.828071    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"]
Jul 28 12:16:42.843812 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.843788    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-model-cache\") pod \"auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") " pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:42.843904 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.843837    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmkpr\" (UniqueName: \"kubernetes.io/projected/26c95989-231f-4a2d-8ef2-1ae4bbaced95-kube-api-access-lmkpr\") pod \"auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") " pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:42.843950 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.843907    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-dshm\") pod \"auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") " pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:42.843986 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.843948    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-tmp-dir\") pod \"auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") " pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:42.844021 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.843987    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-home\") pod \"auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") " pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:42.844056 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.844020    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c95989-231f-4a2d-8ef2-1ae4bbaced95-tls-certs\") pod \"auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") " pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:42.945464 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.945428    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c95989-231f-4a2d-8ef2-1ae4bbaced95-tls-certs\") pod \"auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") " pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:42.945683 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.945512    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-model-cache\") pod \"auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") " pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:42.945683 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.945547    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmkpr\" (UniqueName: \"kubernetes.io/projected/26c95989-231f-4a2d-8ef2-1ae4bbaced95-kube-api-access-lmkpr\") pod \"auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") " pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:42.945683 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.945618    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-dshm\") pod \"auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") " pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:42.945683 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.945644    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-tmp-dir\") pod \"auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") " pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:42.945683 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.945678    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-home\") pod \"auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") " pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:42.946004 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.945983    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-model-cache\") pod \"auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") " pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:42.946111 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.946091    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-home\") pod \"auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") " pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:42.946166 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.946109    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-tmp-dir\") pod \"auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") " pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:42.947708 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.947688    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-dshm\") pod \"auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") " pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:42.947980 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.947960    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c95989-231f-4a2d-8ef2-1ae4bbaced95-tls-certs\") pod \"auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") " pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:42.953158 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:42.953137    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmkpr\" (UniqueName: \"kubernetes.io/projected/26c95989-231f-4a2d-8ef2-1ae4bbaced95-kube-api-access-lmkpr\") pod \"auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") " pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:43.122053 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:43.121971    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:43.253351 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:43.253325    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"]
Jul 28 12:16:43.255271 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:16:43.255243    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26c95989_231f_4a2d_8ef2_1ae4bbaced95.slice/crio-3ea709fc16a1f651647a4ac39c03c450559cfe31de01b4bf32bdf8f08181f8ae WatchSource:0}: Error finding container 3ea709fc16a1f651647a4ac39c03c450559cfe31de01b4bf32bdf8f08181f8ae: Status 404 returned error can't find the container with id 3ea709fc16a1f651647a4ac39c03c450559cfe31de01b4bf32bdf8f08181f8ae
Jul 28 12:16:43.542534 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:43.542437    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj" event={"ID":"26c95989-231f-4a2d-8ef2-1ae4bbaced95","Type":"ContainerStarted","Data":"98cf38dc745de1c9bfe7870f80247139043ec01a08004bab1d65d58c95fa03ca"}
Jul 28 12:16:43.542534 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:43.542480    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj" event={"ID":"26c95989-231f-4a2d-8ef2-1ae4bbaced95","Type":"ContainerStarted","Data":"3ea709fc16a1f651647a4ac39c03c450559cfe31de01b4bf32bdf8f08181f8ae"}
Jul 28 12:16:43.561318 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:43.561263    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj" podStartSLOduration=1.561246411 podStartE2EDuration="1.561246411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:16:43.558889236 +0000 UTC m=+2755.417545234" watchObservedRunningTime="2026-07-28 12:16:43.561246411 +0000 UTC m=+2755.419902409"
Jul 28 12:16:46.227560 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:46.227527    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:16:53.122691 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:53.122649    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:53.123085 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:53.122704    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:53.135420 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:53.135388    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:16:53.598116 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:16:53.598083    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:17:07.338420 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:17:07.338383    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:17:34.133703 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:17:34.133659    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:17:56.031652 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:17:56.031617    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:18:13.530365 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:18:13.530336    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:18:35.327390 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:18:35.327353    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:18:53.727938 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:18:53.727898    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:19:13.822212 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:19:13.822171    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:19:29.125915 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:19:29.125872    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:19:50.928185 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:19:50.928153    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:20:08.726845 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:20:08.726807    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:20:30.530942 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:20:30.530905    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:20:45.621404 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:20:45.621369    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:20:48.851740 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:20:48.851710    2561 scope.go:117] "RemoveContainer" containerID="5a86c8346f180d0569b55b3bc50e121c81d9939fd3cf38cc21c2b2da45503b90"
Jul 28 12:21:12.022017 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:12.021979    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:21:15.186860 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:15.186830    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"]
Jul 28 12:21:28.324468 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:28.324430    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:21:30.205694 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.205633    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj" podUID="26c95989-231f-4a2d-8ef2-1ae4bbaced95" containerName="main" containerID="cri-o://98cf38dc745de1c9bfe7870f80247139043ec01a08004bab1d65d58c95fa03ca" gracePeriod=45
Jul 28 12:21:30.447811 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.447787    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:21:30.520828 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.520757    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-home\") pod \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") "
Jul 28 12:21:30.520975 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.520859    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmkpr\" (UniqueName: \"kubernetes.io/projected/26c95989-231f-4a2d-8ef2-1ae4bbaced95-kube-api-access-lmkpr\") pod \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") "
Jul 28 12:21:30.520975 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.520894    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c95989-231f-4a2d-8ef2-1ae4bbaced95-tls-certs\") pod \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") "
Jul 28 12:21:30.520975 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.520932    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-model-cache\") pod \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") "
Jul 28 12:21:30.520975 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.520954    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-dshm\") pod \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") "
Jul 28 12:21:30.521218 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.521002    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-tmp-dir\") pod \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\" (UID: \"26c95989-231f-4a2d-8ef2-1ae4bbaced95\") "
Jul 28 12:21:30.521218 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.521036    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-home" (OuterVolumeSpecName: "home") pod "26c95989-231f-4a2d-8ef2-1ae4bbaced95" (UID: "26c95989-231f-4a2d-8ef2-1ae4bbaced95"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:21:30.521218 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.521189    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-model-cache" (OuterVolumeSpecName: "model-cache") pod "26c95989-231f-4a2d-8ef2-1ae4bbaced95" (UID: "26c95989-231f-4a2d-8ef2-1ae4bbaced95"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:21:30.521385 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.521266    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-home\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:21:30.521385 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.521284    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-model-cache\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:21:30.521385 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.521344    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "26c95989-231f-4a2d-8ef2-1ae4bbaced95" (UID: "26c95989-231f-4a2d-8ef2-1ae4bbaced95"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:21:30.523154 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.523128    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-dshm" (OuterVolumeSpecName: "dshm") pod "26c95989-231f-4a2d-8ef2-1ae4bbaced95" (UID: "26c95989-231f-4a2d-8ef2-1ae4bbaced95"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:21:30.523260 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.523178    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c95989-231f-4a2d-8ef2-1ae4bbaced95-kube-api-access-lmkpr" (OuterVolumeSpecName: "kube-api-access-lmkpr") pod "26c95989-231f-4a2d-8ef2-1ae4bbaced95" (UID: "26c95989-231f-4a2d-8ef2-1ae4bbaced95"). InnerVolumeSpecName "kube-api-access-lmkpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:21:30.523260 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.523192    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c95989-231f-4a2d-8ef2-1ae4bbaced95-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26c95989-231f-4a2d-8ef2-1ae4bbaced95" (UID: "26c95989-231f-4a2d-8ef2-1ae4bbaced95"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:21:30.622116 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.622083    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-tmp-dir\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:21:30.622116 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.622113    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmkpr\" (UniqueName: \"kubernetes.io/projected/26c95989-231f-4a2d-8ef2-1ae4bbaced95-kube-api-access-lmkpr\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:21:30.622268 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.622122    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26c95989-231f-4a2d-8ef2-1ae4bbaced95-tls-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:21:30.622268 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.622130    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26c95989-231f-4a2d-8ef2-1ae4bbaced95-dshm\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:21:30.812067 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.812033    2561 generic.go:356] "Generic (PLEG): container finished" podID="26c95989-231f-4a2d-8ef2-1ae4bbaced95" containerID="98cf38dc745de1c9bfe7870f80247139043ec01a08004bab1d65d58c95fa03ca" exitCode=0
Jul 28 12:21:30.812230 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.812081    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"
Jul 28 12:21:30.812230 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.812121    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj" event={"ID":"26c95989-231f-4a2d-8ef2-1ae4bbaced95","Type":"ContainerDied","Data":"98cf38dc745de1c9bfe7870f80247139043ec01a08004bab1d65d58c95fa03ca"}
Jul 28 12:21:30.812230 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.812164    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj" event={"ID":"26c95989-231f-4a2d-8ef2-1ae4bbaced95","Type":"ContainerDied","Data":"3ea709fc16a1f651647a4ac39c03c450559cfe31de01b4bf32bdf8f08181f8ae"}
Jul 28 12:21:30.812230 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.812182    2561 scope.go:117] "RemoveContainer" containerID="98cf38dc745de1c9bfe7870f80247139043ec01a08004bab1d65d58c95fa03ca"
Jul 28 12:21:30.820588 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.820558    2561 scope.go:117] "RemoveContainer" containerID="98cf38dc745de1c9bfe7870f80247139043ec01a08004bab1d65d58c95fa03ca"
Jul 28 12:21:30.820854 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:21:30.820835    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98cf38dc745de1c9bfe7870f80247139043ec01a08004bab1d65d58c95fa03ca\": container with ID starting with 98cf38dc745de1c9bfe7870f80247139043ec01a08004bab1d65d58c95fa03ca not found: ID does not exist" containerID="98cf38dc745de1c9bfe7870f80247139043ec01a08004bab1d65d58c95fa03ca"
Jul 28 12:21:30.820939 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.820861    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98cf38dc745de1c9bfe7870f80247139043ec01a08004bab1d65d58c95fa03ca"} err="failed to get container status \"98cf38dc745de1c9bfe7870f80247139043ec01a08004bab1d65d58c95fa03ca\": rpc error: code = NotFound desc = could not find container \"98cf38dc745de1c9bfe7870f80247139043ec01a08004bab1d65d58c95fa03ca\": container with ID starting with 98cf38dc745de1c9bfe7870f80247139043ec01a08004bab1d65d58c95fa03ca not found: ID does not exist"
Jul 28 12:21:30.828709 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.828686    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"]
Jul 28 12:21:30.832335 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:30.832315    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4d5cfe/auth-test-model-4d5cfe-kserve-6b54cd7475-bzljj"]
Jul 28 12:21:32.546745 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:32.546715    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26c95989-231f-4a2d-8ef2-1ae4bbaced95" path="/var/lib/kubelet/pods/26c95989-231f-4a2d-8ef2-1ae4bbaced95/volumes"
Jul 28 12:21:37.434397 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:37.434364    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:21:49.706466 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.706419    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"]
Jul 28 12:21:49.706917 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.706835    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26c95989-231f-4a2d-8ef2-1ae4bbaced95" containerName="main"
Jul 28 12:21:49.706917 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.706849    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="26c95989-231f-4a2d-8ef2-1ae4bbaced95" containerName="main"
Jul 28 12:21:49.706917 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.706910    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="26c95989-231f-4a2d-8ef2-1ae4bbaced95" containerName="main"
Jul 28 12:21:49.710044 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.710026    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:21:49.712160 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.712140    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e7af0c\"/\"openshift-service-ca.crt\""
Jul 28 12:21:49.713019 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.713000    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e7af0c\"/\"kube-root-ca.crt\""
Jul 28 12:21:49.713116 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.713003    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e7af0c\"/\"default-dockercfg-sn6d8\""
Jul 28 12:21:49.713282 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.713268    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e7af0c\"/\"sub-test-model-e7af0c-kserve-self-signed-certs\""
Jul 28 12:21:49.718154 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.718132    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"]
Jul 28 12:21:49.787772 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.787739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-home\") pod \"sub-test-model-e7af0c-kserve-56cb4779-xnwxj\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") " pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:21:49.787939 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.787787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-model-cache\") pod \"sub-test-model-e7af0c-kserve-56cb4779-xnwxj\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") " pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:21:49.787939 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.787812    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfjr8\" (UniqueName: \"kubernetes.io/projected/564b1142-a30f-43e4-906c-223e71dca392-kube-api-access-kfjr8\") pod \"sub-test-model-e7af0c-kserve-56cb4779-xnwxj\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") " pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:21:49.787939 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.787838    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/564b1142-a30f-43e4-906c-223e71dca392-tls-certs\") pod \"sub-test-model-e7af0c-kserve-56cb4779-xnwxj\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") " pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:21:49.787939 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.787864    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-tmp-dir\") pod \"sub-test-model-e7af0c-kserve-56cb4779-xnwxj\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") " pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:21:49.788089 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.787962    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-dshm\") pod \"sub-test-model-e7af0c-kserve-56cb4779-xnwxj\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") " pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:21:49.888415 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.888379    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-dshm\") pod \"sub-test-model-e7af0c-kserve-56cb4779-xnwxj\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") " pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:21:49.888609 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.888435    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-home\") pod \"sub-test-model-e7af0c-kserve-56cb4779-xnwxj\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") " pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:21:49.888609 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.888466    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-model-cache\") pod \"sub-test-model-e7af0c-kserve-56cb4779-xnwxj\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") " pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:21:49.888609 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.888484    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfjr8\" (UniqueName: \"kubernetes.io/projected/564b1142-a30f-43e4-906c-223e71dca392-kube-api-access-kfjr8\") pod \"sub-test-model-e7af0c-kserve-56cb4779-xnwxj\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") " pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:21:49.888609 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.888500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/564b1142-a30f-43e4-906c-223e71dca392-tls-certs\") pod \"sub-test-model-e7af0c-kserve-56cb4779-xnwxj\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") " pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:21:49.888609 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.888528    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-tmp-dir\") pod \"sub-test-model-e7af0c-kserve-56cb4779-xnwxj\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") " pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:21:49.888910 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.888883    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-model-cache\") pod \"sub-test-model-e7af0c-kserve-56cb4779-xnwxj\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") " pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:21:49.888977 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.888948    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-tmp-dir\") pod \"sub-test-model-e7af0c-kserve-56cb4779-xnwxj\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") " pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:21:49.888977 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.888944    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-home\") pod \"sub-test-model-e7af0c-kserve-56cb4779-xnwxj\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") " pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:21:49.890723 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.890698    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-dshm\") pod \"sub-test-model-e7af0c-kserve-56cb4779-xnwxj\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") " pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:21:49.890965 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.890945    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/564b1142-a30f-43e4-906c-223e71dca392-tls-certs\") pod \"sub-test-model-e7af0c-kserve-56cb4779-xnwxj\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") " pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:21:49.897683 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:49.897661    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfjr8\" (UniqueName: \"kubernetes.io/projected/564b1142-a30f-43e4-906c-223e71dca392-kube-api-access-kfjr8\") pod \"sub-test-model-e7af0c-kserve-56cb4779-xnwxj\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") " pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:21:50.021123 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:50.021039    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:21:50.152348 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:50.152324    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"]
Jul 28 12:21:50.154446 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:21:50.154419    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod564b1142_a30f_43e4_906c_223e71dca392.slice/crio-b27881a27ae813cd22b1c44de8510e1bcac435fe2939f6081f444e7d84506038 WatchSource:0}: Error finding container b27881a27ae813cd22b1c44de8510e1bcac435fe2939f6081f444e7d84506038: Status 404 returned error can't find the container with id b27881a27ae813cd22b1c44de8510e1bcac435fe2939f6081f444e7d84506038
Jul 28 12:21:50.902758 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:50.902726    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj" event={"ID":"564b1142-a30f-43e4-906c-223e71dca392","Type":"ContainerStarted","Data":"79e37f66b2369e5de4fb818f885642ebe86a730102526c946c7b37eaffb3617a"}
Jul 28 12:21:50.903110 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:50.902769    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj" event={"ID":"564b1142-a30f-43e4-906c-223e71dca392","Type":"ContainerStarted","Data":"b27881a27ae813cd22b1c44de8510e1bcac435fe2939f6081f444e7d84506038"}
Jul 28 12:21:50.920431 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:50.920383    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj" podStartSLOduration=1.920367683 podStartE2EDuration="1.920367683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:21:50.919716993 +0000 UTC m=+3062.778372984" watchObservedRunningTime="2026-07-28 12:21:50.920367683 +0000 UTC m=+3062.779023682"
Jul 28 12:21:53.230964 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:21:53.230919    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:22:00.022144 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:22:00.022113    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:22:00.022573 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:22:00.022368    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:22:00.034709 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:22:00.034688    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:22:00.960596 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:22:00.960552    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:22:24.818861 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:22:24.818828    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:22:42.435640 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:22:42.435608    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:23:03.424876 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:03.424838    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:23:09.984403 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:09.984371    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"]
Jul 28 12:23:18.421485 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.421402    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"]
Jul 28 12:23:18.424790 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.424775    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:18.427260 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.427239    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4d5cfe\"/\"openshift-service-ca.crt\""
Jul 28 12:23:18.427445 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.427424    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4d5cfe\"/\"default-dockercfg-8gj6v\""
Jul 28 12:23:18.427953 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.427934    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4d5cfe\"/\"rate-test-model-4d5cfe-kserve-self-signed-certs\""
Jul 28 12:23:18.428025 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.428009    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4d5cfe\"/\"kube-root-ca.crt\""
Jul 28 12:23:18.443602 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.443556    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"]
Jul 28 12:23:18.556017 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.555990    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mjz5\" (UniqueName: \"kubernetes.io/projected/62d531ee-110c-44aa-9e18-b076d037634b-kube-api-access-2mjz5\") pod \"rate-test-model-4d5cfe-kserve-655558df6c-g5hc7\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") " pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:18.556180 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.556057    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-tmp-dir\") pod \"rate-test-model-4d5cfe-kserve-655558df6c-g5hc7\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") " pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:18.556180 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.556089    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-home\") pod \"rate-test-model-4d5cfe-kserve-655558df6c-g5hc7\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") " pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:18.556180 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.556104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-dshm\") pod \"rate-test-model-4d5cfe-kserve-655558df6c-g5hc7\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") " pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:18.556180 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.556120    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62d531ee-110c-44aa-9e18-b076d037634b-tls-certs\") pod \"rate-test-model-4d5cfe-kserve-655558df6c-g5hc7\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") " pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:18.556319 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.556206    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-model-cache\") pod \"rate-test-model-4d5cfe-kserve-655558df6c-g5hc7\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") " pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:18.657550 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.657516    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-home\") pod \"rate-test-model-4d5cfe-kserve-655558df6c-g5hc7\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") " pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:18.657550 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.657546    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-dshm\") pod \"rate-test-model-4d5cfe-kserve-655558df6c-g5hc7\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") " pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:18.657762 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.657565    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62d531ee-110c-44aa-9e18-b076d037634b-tls-certs\") pod \"rate-test-model-4d5cfe-kserve-655558df6c-g5hc7\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") " pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:18.657762 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.657604    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-model-cache\") pod \"rate-test-model-4d5cfe-kserve-655558df6c-g5hc7\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") " pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:18.657762 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.657736    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mjz5\" (UniqueName: \"kubernetes.io/projected/62d531ee-110c-44aa-9e18-b076d037634b-kube-api-access-2mjz5\") pod \"rate-test-model-4d5cfe-kserve-655558df6c-g5hc7\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") " pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:18.657931 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.657836    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-tmp-dir\") pod \"rate-test-model-4d5cfe-kserve-655558df6c-g5hc7\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") " pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:18.657931 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.657907    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-home\") pod \"rate-test-model-4d5cfe-kserve-655558df6c-g5hc7\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") " pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:18.658047 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.657987    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-model-cache\") pod \"rate-test-model-4d5cfe-kserve-655558df6c-g5hc7\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") " pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:18.658186 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.658157    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-tmp-dir\") pod \"rate-test-model-4d5cfe-kserve-655558df6c-g5hc7\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") " pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:18.659918 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.659898    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-dshm\") pod \"rate-test-model-4d5cfe-kserve-655558df6c-g5hc7\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") " pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:18.660015 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.660000    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62d531ee-110c-44aa-9e18-b076d037634b-tls-certs\") pod \"rate-test-model-4d5cfe-kserve-655558df6c-g5hc7\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") " pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:18.667476 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.667458    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mjz5\" (UniqueName: \"kubernetes.io/projected/62d531ee-110c-44aa-9e18-b076d037634b-kube-api-access-2mjz5\") pod \"rate-test-model-4d5cfe-kserve-655558df6c-g5hc7\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") " pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:18.733957 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.733904    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:18.868408 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:18.868377    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"]
Jul 28 12:23:18.871945 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:23:18.871909    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62d531ee_110c_44aa_9e18_b076d037634b.slice/crio-eb9967d148d3e2b20acd99636d4580582d765946ed3208f5ae72150aebb4e268 WatchSource:0}: Error finding container eb9967d148d3e2b20acd99636d4580582d765946ed3208f5ae72150aebb4e268: Status 404 returned error can't find the container with id eb9967d148d3e2b20acd99636d4580582d765946ed3208f5ae72150aebb4e268
Jul 28 12:23:19.292985 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:19.292943    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7" event={"ID":"62d531ee-110c-44aa-9e18-b076d037634b","Type":"ContainerStarted","Data":"6a06078a793ec6685d4e65fb75fba1c5f8f3406c9fab5e409681a992a2b0259b"}
Jul 28 12:23:19.293171 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:19.292992    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7" event={"ID":"62d531ee-110c-44aa-9e18-b076d037634b","Type":"ContainerStarted","Data":"eb9967d148d3e2b20acd99636d4580582d765946ed3208f5ae72150aebb4e268"}
Jul 28 12:23:19.313211 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:19.313163    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7" podStartSLOduration=1.313142838 podStartE2EDuration="1.313142838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:23:19.311636015 +0000 UTC m=+3151.170292011" watchObservedRunningTime="2026-07-28 12:23:19.313142838 +0000 UTC m=+3151.171798838"
Jul 28 12:23:22.048179 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:22.048136    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:23:25.002809 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.002760    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj" podUID="564b1142-a30f-43e4-906c-223e71dca392" containerName="main" containerID="cri-o://79e37f66b2369e5de4fb818f885642ebe86a730102526c946c7b37eaffb3617a" gracePeriod=45
Jul 28 12:23:25.255746 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.255691    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:23:25.322475 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.322440    2561 generic.go:356] "Generic (PLEG): container finished" podID="564b1142-a30f-43e4-906c-223e71dca392" containerID="79e37f66b2369e5de4fb818f885642ebe86a730102526c946c7b37eaffb3617a" exitCode=0
Jul 28 12:23:25.322671 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.322490    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"
Jul 28 12:23:25.322671 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.322488    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj" event={"ID":"564b1142-a30f-43e4-906c-223e71dca392","Type":"ContainerDied","Data":"79e37f66b2369e5de4fb818f885642ebe86a730102526c946c7b37eaffb3617a"}
Jul 28 12:23:25.322671 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.322618    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj" event={"ID":"564b1142-a30f-43e4-906c-223e71dca392","Type":"ContainerDied","Data":"b27881a27ae813cd22b1c44de8510e1bcac435fe2939f6081f444e7d84506038"}
Jul 28 12:23:25.322671 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.322640    2561 scope.go:117] "RemoveContainer" containerID="79e37f66b2369e5de4fb818f885642ebe86a730102526c946c7b37eaffb3617a"
Jul 28 12:23:25.330574 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.330555    2561 scope.go:117] "RemoveContainer" containerID="79e37f66b2369e5de4fb818f885642ebe86a730102526c946c7b37eaffb3617a"
Jul 28 12:23:25.330842 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:23:25.330822    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e37f66b2369e5de4fb818f885642ebe86a730102526c946c7b37eaffb3617a\": container with ID starting with 79e37f66b2369e5de4fb818f885642ebe86a730102526c946c7b37eaffb3617a not found: ID does not exist" containerID="79e37f66b2369e5de4fb818f885642ebe86a730102526c946c7b37eaffb3617a"
Jul 28 12:23:25.330893 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.330851    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e37f66b2369e5de4fb818f885642ebe86a730102526c946c7b37eaffb3617a"} err="failed to get container status \"79e37f66b2369e5de4fb818f885642ebe86a730102526c946c7b37eaffb3617a\": rpc error: code = NotFound desc = could not find container \"79e37f66b2369e5de4fb818f885642ebe86a730102526c946c7b37eaffb3617a\": container with ID starting with 79e37f66b2369e5de4fb818f885642ebe86a730102526c946c7b37eaffb3617a not found: ID does not exist"
Jul 28 12:23:25.416990 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.416934    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-dshm\") pod \"564b1142-a30f-43e4-906c-223e71dca392\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") "
Jul 28 12:23:25.416990 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.416991    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-home\") pod \"564b1142-a30f-43e4-906c-223e71dca392\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") "
Jul 28 12:23:25.417270 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.417029    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/564b1142-a30f-43e4-906c-223e71dca392-tls-certs\") pod \"564b1142-a30f-43e4-906c-223e71dca392\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") "
Jul 28 12:23:25.417270 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.417079    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfjr8\" (UniqueName: \"kubernetes.io/projected/564b1142-a30f-43e4-906c-223e71dca392-kube-api-access-kfjr8\") pod \"564b1142-a30f-43e4-906c-223e71dca392\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") "
Jul 28 12:23:25.417270 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.417125    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-model-cache\") pod \"564b1142-a30f-43e4-906c-223e71dca392\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") "
Jul 28 12:23:25.417270 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.417177    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-tmp-dir\") pod \"564b1142-a30f-43e4-906c-223e71dca392\" (UID: \"564b1142-a30f-43e4-906c-223e71dca392\") "
Jul 28 12:23:25.417489 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.417322    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-home" (OuterVolumeSpecName: "home") pod "564b1142-a30f-43e4-906c-223e71dca392" (UID: "564b1142-a30f-43e4-906c-223e71dca392"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:23:25.417489 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.417434    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-model-cache" (OuterVolumeSpecName: "model-cache") pod "564b1142-a30f-43e4-906c-223e71dca392" (UID: "564b1142-a30f-43e4-906c-223e71dca392"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:23:25.417624 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.417489    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-home\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:23:25.417624 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.417494    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "564b1142-a30f-43e4-906c-223e71dca392" (UID: "564b1142-a30f-43e4-906c-223e71dca392"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:23:25.419377 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.419346    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/564b1142-a30f-43e4-906c-223e71dca392-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "564b1142-a30f-43e4-906c-223e71dca392" (UID: "564b1142-a30f-43e4-906c-223e71dca392"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:23:25.419484 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.419379    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-dshm" (OuterVolumeSpecName: "dshm") pod "564b1142-a30f-43e4-906c-223e71dca392" (UID: "564b1142-a30f-43e4-906c-223e71dca392"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:23:25.419484 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.419441    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564b1142-a30f-43e4-906c-223e71dca392-kube-api-access-kfjr8" (OuterVolumeSpecName: "kube-api-access-kfjr8") pod "564b1142-a30f-43e4-906c-223e71dca392" (UID: "564b1142-a30f-43e4-906c-223e71dca392"). InnerVolumeSpecName "kube-api-access-kfjr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:23:25.518103 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.518023    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/564b1142-a30f-43e4-906c-223e71dca392-tls-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:23:25.518103 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.518050    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfjr8\" (UniqueName: \"kubernetes.io/projected/564b1142-a30f-43e4-906c-223e71dca392-kube-api-access-kfjr8\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:23:25.518103 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.518060    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-model-cache\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:23:25.518103 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.518069    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-tmp-dir\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:23:25.518103 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.518076    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/564b1142-a30f-43e4-906c-223e71dca392-dshm\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:23:25.650628 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.650574    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"]
Jul 28 12:23:25.651995 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:25.651975    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e7af0c/sub-test-model-e7af0c-kserve-56cb4779-xnwxj"]
Jul 28 12:23:26.546919 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:26.546881    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="564b1142-a30f-43e4-906c-223e71dca392" path="/var/lib/kubelet/pods/564b1142-a30f-43e4-906c-223e71dca392/volumes"
Jul 28 12:23:28.734662 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:28.734628    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:28.734662 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:28.734668    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:28.747307 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:28.747284    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:29.351529 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:29.351500    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:23:46.033764 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:23:46.033725    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:24:18.531445 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:24:18.531408    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:24:31.433085 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:24:31.433044    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:24:49.230368 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:24:49.230280    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:24:53.890014 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:24:53.889977    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"]
Jul 28 12:25:05.522362 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:05.522323    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:25:08.909796 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:08.909745    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7" podUID="62d531ee-110c-44aa-9e18-b076d037634b" containerName="main" containerID="cri-o://6a06078a793ec6685d4e65fb75fba1c5f8f3406c9fab5e409681a992a2b0259b" gracePeriod=45
Jul 28 12:25:09.154890 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.154867    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:25:09.311762 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.311732    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62d531ee-110c-44aa-9e18-b076d037634b-tls-certs\") pod \"62d531ee-110c-44aa-9e18-b076d037634b\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") "
Jul 28 12:25:09.311914 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.311776    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-dshm\") pod \"62d531ee-110c-44aa-9e18-b076d037634b\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") "
Jul 28 12:25:09.311914 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.311811    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-model-cache\") pod \"62d531ee-110c-44aa-9e18-b076d037634b\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") "
Jul 28 12:25:09.311914 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.311889    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-tmp-dir\") pod \"62d531ee-110c-44aa-9e18-b076d037634b\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") "
Jul 28 12:25:09.312054 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.311935    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mjz5\" (UniqueName: \"kubernetes.io/projected/62d531ee-110c-44aa-9e18-b076d037634b-kube-api-access-2mjz5\") pod \"62d531ee-110c-44aa-9e18-b076d037634b\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") "
Jul 28 12:25:09.312054 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.311970    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-home\") pod \"62d531ee-110c-44aa-9e18-b076d037634b\" (UID: \"62d531ee-110c-44aa-9e18-b076d037634b\") "
Jul 28 12:25:09.312203 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.312151    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62d531ee-110c-44aa-9e18-b076d037634b" (UID: "62d531ee-110c-44aa-9e18-b076d037634b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:25:09.312203 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.312162    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-model-cache" (OuterVolumeSpecName: "model-cache") pod "62d531ee-110c-44aa-9e18-b076d037634b" (UID: "62d531ee-110c-44aa-9e18-b076d037634b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:25:09.312363 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.312299    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-home" (OuterVolumeSpecName: "home") pod "62d531ee-110c-44aa-9e18-b076d037634b" (UID: "62d531ee-110c-44aa-9e18-b076d037634b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:25:09.312363 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.312312    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-model-cache\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:25:09.312457 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.312375    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-tmp-dir\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:25:09.313998 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.313979    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-dshm" (OuterVolumeSpecName: "dshm") pod "62d531ee-110c-44aa-9e18-b076d037634b" (UID: "62d531ee-110c-44aa-9e18-b076d037634b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:25:09.314089 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.314037    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d531ee-110c-44aa-9e18-b076d037634b-kube-api-access-2mjz5" (OuterVolumeSpecName: "kube-api-access-2mjz5") pod "62d531ee-110c-44aa-9e18-b076d037634b" (UID: "62d531ee-110c-44aa-9e18-b076d037634b"). InnerVolumeSpecName "kube-api-access-2mjz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:25:09.314089 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.314079    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d531ee-110c-44aa-9e18-b076d037634b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62d531ee-110c-44aa-9e18-b076d037634b" (UID: "62d531ee-110c-44aa-9e18-b076d037634b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:25:09.413453 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.413413    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mjz5\" (UniqueName: \"kubernetes.io/projected/62d531ee-110c-44aa-9e18-b076d037634b-kube-api-access-2mjz5\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:25:09.413453 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.413447    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-home\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:25:09.413453 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.413458    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62d531ee-110c-44aa-9e18-b076d037634b-tls-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:25:09.413728 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.413467    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62d531ee-110c-44aa-9e18-b076d037634b-dshm\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:25:09.783873 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.783786    2561 generic.go:356] "Generic (PLEG): container finished" podID="62d531ee-110c-44aa-9e18-b076d037634b" containerID="6a06078a793ec6685d4e65fb75fba1c5f8f3406c9fab5e409681a992a2b0259b" exitCode=0
Jul 28 12:25:09.783873 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.783825    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"
Jul 28 12:25:09.784110 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.783867    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7" event={"ID":"62d531ee-110c-44aa-9e18-b076d037634b","Type":"ContainerDied","Data":"6a06078a793ec6685d4e65fb75fba1c5f8f3406c9fab5e409681a992a2b0259b"}
Jul 28 12:25:09.784110 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.783914    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7" event={"ID":"62d531ee-110c-44aa-9e18-b076d037634b","Type":"ContainerDied","Data":"eb9967d148d3e2b20acd99636d4580582d765946ed3208f5ae72150aebb4e268"}
Jul 28 12:25:09.784110 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.783936    2561 scope.go:117] "RemoveContainer" containerID="6a06078a793ec6685d4e65fb75fba1c5f8f3406c9fab5e409681a992a2b0259b"
Jul 28 12:25:09.793493 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.793475    2561 scope.go:117] "RemoveContainer" containerID="6a06078a793ec6685d4e65fb75fba1c5f8f3406c9fab5e409681a992a2b0259b"
Jul 28 12:25:09.793783 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:25:09.793763    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a06078a793ec6685d4e65fb75fba1c5f8f3406c9fab5e409681a992a2b0259b\": container with ID starting with 6a06078a793ec6685d4e65fb75fba1c5f8f3406c9fab5e409681a992a2b0259b not found: ID does not exist" containerID="6a06078a793ec6685d4e65fb75fba1c5f8f3406c9fab5e409681a992a2b0259b"
Jul 28 12:25:09.793888 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.793788    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a06078a793ec6685d4e65fb75fba1c5f8f3406c9fab5e409681a992a2b0259b"} err="failed to get container status \"6a06078a793ec6685d4e65fb75fba1c5f8f3406c9fab5e409681a992a2b0259b\": rpc error: code = NotFound desc = could not find container \"6a06078a793ec6685d4e65fb75fba1c5f8f3406c9fab5e409681a992a2b0259b\": container with ID starting with 6a06078a793ec6685d4e65fb75fba1c5f8f3406c9fab5e409681a992a2b0259b not found: ID does not exist"
Jul 28 12:25:09.804561 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.804539    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"]
Jul 28 12:25:09.809845 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:09.809820    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4d5cfe/rate-test-model-4d5cfe-kserve-655558df6c-g5hc7"]
Jul 28 12:25:10.546662 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:10.546630    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62d531ee-110c-44aa-9e18-b076d037634b" path="/var/lib/kubelet/pods/62d531ee-110c-44aa-9e18-b076d037634b/volumes"
Jul 28 12:25:18.756904 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.756868    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv"]
Jul 28 12:25:18.757287 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.757199    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="564b1142-a30f-43e4-906c-223e71dca392" containerName="main"
Jul 28 12:25:18.757287 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.757210    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="564b1142-a30f-43e4-906c-223e71dca392" containerName="main"
Jul 28 12:25:18.757287 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.757217    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d531ee-110c-44aa-9e18-b076d037634b" containerName="main"
Jul 28 12:25:18.757287 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.757222    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d531ee-110c-44aa-9e18-b076d037634b" containerName="main"
Jul 28 12:25:18.757427 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.757299    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d531ee-110c-44aa-9e18-b076d037634b" containerName="main"
Jul 28 12:25:18.757427 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.757311    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="564b1142-a30f-43e4-906c-223e71dca392" containerName="main"
Jul 28 12:25:18.761362 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.761343    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv"
Jul 28 12:25:18.763440 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.763418    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ba0125\""
Jul 28 12:25:18.763592 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.763440    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ba0125-dockercfg-mvzm6\""
Jul 28 12:25:18.768060 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.768034    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv"]
Jul 28 12:25:18.795362 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.795318    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49"]
Jul 28 12:25:18.798641 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.798621    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49"
Jul 28 12:25:18.806979 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.806953    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49"]
Jul 28 12:25:18.898720 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.898682    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbd0f8c3-2b81-4319-b624-5104697eb0ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv\" (UID: \"cbd0f8c3-2b81-4319-b624-5104697eb0ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv"
Jul 28 12:25:18.898720 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.898722    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8474023-b495-4710-9062-6e008ff5c712-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49\" (UID: \"b8474023-b495-4710-9062-6e008ff5c712\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49"
Jul 28 12:25:18.898916 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.898756    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqhnh\" (UniqueName: \"kubernetes.io/projected/b8474023-b495-4710-9062-6e008ff5c712-kube-api-access-bqhnh\") pod \"payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49\" (UID: \"b8474023-b495-4710-9062-6e008ff5c712\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49"
Jul 28 12:25:18.898916 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.898854    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwtg5\" (UniqueName: \"kubernetes.io/projected/cbd0f8c3-2b81-4319-b624-5104697eb0ee-kube-api-access-zwtg5\") pod \"payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv\" (UID: \"cbd0f8c3-2b81-4319-b624-5104697eb0ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv"
Jul 28 12:25:18.999221 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.999191    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwtg5\" (UniqueName: \"kubernetes.io/projected/cbd0f8c3-2b81-4319-b624-5104697eb0ee-kube-api-access-zwtg5\") pod \"payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv\" (UID: \"cbd0f8c3-2b81-4319-b624-5104697eb0ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv"
Jul 28 12:25:18.999396 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.999250    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbd0f8c3-2b81-4319-b624-5104697eb0ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv\" (UID: \"cbd0f8c3-2b81-4319-b624-5104697eb0ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv"
Jul 28 12:25:18.999396 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.999270    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8474023-b495-4710-9062-6e008ff5c712-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49\" (UID: \"b8474023-b495-4710-9062-6e008ff5c712\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49"
Jul 28 12:25:18.999396 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.999299    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqhnh\" (UniqueName: \"kubernetes.io/projected/b8474023-b495-4710-9062-6e008ff5c712-kube-api-access-bqhnh\") pod \"payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49\" (UID: \"b8474023-b495-4710-9062-6e008ff5c712\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49"
Jul 28 12:25:19.000020 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.999997    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8474023-b495-4710-9062-6e008ff5c712-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49\" (UID: \"b8474023-b495-4710-9062-6e008ff5c712\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49"
Jul 28 12:25:19.000115 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:18.999999    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbd0f8c3-2b81-4319-b624-5104697eb0ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv\" (UID: \"cbd0f8c3-2b81-4319-b624-5104697eb0ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv"
Jul 28 12:25:19.008480 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:19.008416    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwtg5\" (UniqueName: \"kubernetes.io/projected/cbd0f8c3-2b81-4319-b624-5104697eb0ee-kube-api-access-zwtg5\") pod \"payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv\" (UID: \"cbd0f8c3-2b81-4319-b624-5104697eb0ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv"
Jul 28 12:25:19.008480 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:19.008429    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqhnh\" (UniqueName: \"kubernetes.io/projected/b8474023-b495-4710-9062-6e008ff5c712-kube-api-access-bqhnh\") pod \"payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49\" (UID: \"b8474023-b495-4710-9062-6e008ff5c712\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49"
Jul 28 12:25:19.073408 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:19.073385    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv"
Jul 28 12:25:19.109555 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:19.109532    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49"
Jul 28 12:25:19.216094 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:19.216011    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv"]
Jul 28 12:25:19.218773 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:25:19.218740    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd0f8c3_2b81_4319_b624_5104697eb0ee.slice/crio-89a01ba98a5748ac6e03f5422c21f1167afb96fb867462958de58f3f8ac40f3e WatchSource:0}: Error finding container 89a01ba98a5748ac6e03f5422c21f1167afb96fb867462958de58f3f8ac40f3e: Status 404 returned error can't find the container with id 89a01ba98a5748ac6e03f5422c21f1167afb96fb867462958de58f3f8ac40f3e
Jul 28 12:25:19.253616 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:19.253571    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49"]
Jul 28 12:25:19.256124 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:25:19.256102    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8474023_b495_4710_9062_6e008ff5c712.slice/crio-147c31432b94f6b15b7d3ebb8bfa68f53ffbe05add546fd2b74343beaf9a4f96 WatchSource:0}: Error finding container 147c31432b94f6b15b7d3ebb8bfa68f53ffbe05add546fd2b74343beaf9a4f96: Status 404 returned error can't find the container with id 147c31432b94f6b15b7d3ebb8bfa68f53ffbe05add546fd2b74343beaf9a4f96
Jul 28 12:25:19.835825 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:19.835788    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49" event={"ID":"b8474023-b495-4710-9062-6e008ff5c712","Type":"ContainerStarted","Data":"aeaf406ecf0c1808e971df51ed959ae7b32751894f9a44bbfeb9922149db12fd"}
Jul 28 12:25:19.836219 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:19.835867    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49" event={"ID":"b8474023-b495-4710-9062-6e008ff5c712","Type":"ContainerStarted","Data":"147c31432b94f6b15b7d3ebb8bfa68f53ffbe05add546fd2b74343beaf9a4f96"}
Jul 28 12:25:19.836219 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:19.835915    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49"
Jul 28 12:25:19.837325 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:19.837301    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv" event={"ID":"cbd0f8c3-2b81-4319-b624-5104697eb0ee","Type":"ContainerStarted","Data":"de1a65ea8eebfebfa4eafbc401e31e2c96d9a2de0952538cb0d8e0643c2b1f0d"}
Jul 28 12:25:19.837434 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:19.837347    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv" event={"ID":"cbd0f8c3-2b81-4319-b624-5104697eb0ee","Type":"ContainerStarted","Data":"89a01ba98a5748ac6e03f5422c21f1167afb96fb867462958de58f3f8ac40f3e"}
Jul 28 12:25:19.837434 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:19.837371    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv"
Jul 28 12:25:19.852052 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:19.852010    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49" podStartSLOduration=1.851997934 podStartE2EDuration="1.851997934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:25:19.8499942 +0000 UTC m=+3271.708650199" watchObservedRunningTime="2026-07-28 12:25:19.851997934 +0000 UTC m=+3271.710653933"
Jul 28 12:25:19.865228 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:19.865190    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv" podStartSLOduration=1.8651733 podStartE2EDuration="1.8651733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:25:19.864048181 +0000 UTC m=+3271.722704205" watchObservedRunningTime="2026-07-28 12:25:19.8651733 +0000 UTC m=+3271.723829298"
Jul 28 12:25:26.923281 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:26.923246    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:25:30.844402 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:30.844369    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49"
Jul 28 12:25:30.844912 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:30.844458    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv"
Jul 28 12:25:36.866920 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:36.866884    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"]
Jul 28 12:25:36.870264 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:36.870232    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"
Jul 28 12:25:36.872479 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:36.872452    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a4d289-serving-cert\""
Jul 28 12:25:36.882179 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:36.882154    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"]
Jul 28 12:25:36.945924 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:36.945889    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b39f2b10-84e5-433d-a9ee-c4e68f9f7650-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg\" (UID: \"b39f2b10-84e5-433d-a9ee-c4e68f9f7650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"
Jul 28 12:25:36.946069 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:36.945947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhlp5\" (UniqueName: \"kubernetes.io/projected/b39f2b10-84e5-433d-a9ee-c4e68f9f7650-kube-api-access-fhlp5\") pod \"maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg\" (UID: \"b39f2b10-84e5-433d-a9ee-c4e68f9f7650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"
Jul 28 12:25:36.946069 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:36.946015    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b39f2b10-84e5-433d-a9ee-c4e68f9f7650-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg\" (UID: \"b39f2b10-84e5-433d-a9ee-c4e68f9f7650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"
Jul 28 12:25:37.047070 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:37.047041    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhlp5\" (UniqueName: \"kubernetes.io/projected/b39f2b10-84e5-433d-a9ee-c4e68f9f7650-kube-api-access-fhlp5\") pod \"maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg\" (UID: \"b39f2b10-84e5-433d-a9ee-c4e68f9f7650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"
Jul 28 12:25:37.047236 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:37.047074    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b39f2b10-84e5-433d-a9ee-c4e68f9f7650-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg\" (UID: \"b39f2b10-84e5-433d-a9ee-c4e68f9f7650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"
Jul 28 12:25:37.047236 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:37.047123    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b39f2b10-84e5-433d-a9ee-c4e68f9f7650-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg\" (UID: \"b39f2b10-84e5-433d-a9ee-c4e68f9f7650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"
Jul 28 12:25:37.047712 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:37.047691    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b39f2b10-84e5-433d-a9ee-c4e68f9f7650-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg\" (UID: \"b39f2b10-84e5-433d-a9ee-c4e68f9f7650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"
Jul 28 12:25:37.049456 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:37.049437    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b39f2b10-84e5-433d-a9ee-c4e68f9f7650-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg\" (UID: \"b39f2b10-84e5-433d-a9ee-c4e68f9f7650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"
Jul 28 12:25:37.054781 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:37.054761    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhlp5\" (UniqueName: \"kubernetes.io/projected/b39f2b10-84e5-433d-a9ee-c4e68f9f7650-kube-api-access-fhlp5\") pod \"maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg\" (UID: \"b39f2b10-84e5-433d-a9ee-c4e68f9f7650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"
Jul 28 12:25:37.184909 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:37.184844    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"
Jul 28 12:25:37.368404 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:37.368375    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"]
Jul 28 12:25:37.369938 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:25:37.369914    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39f2b10_84e5_433d_a9ee_c4e68f9f7650.slice/crio-dab7e73cdb83ab6cd296ac7526bd3133ff98f3bec0e14b2c709e391dcdab0506 WatchSource:0}: Error finding container dab7e73cdb83ab6cd296ac7526bd3133ff98f3bec0e14b2c709e391dcdab0506: Status 404 returned error can't find the container with id dab7e73cdb83ab6cd296ac7526bd3133ff98f3bec0e14b2c709e391dcdab0506
Jul 28 12:25:37.371433 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:37.371412    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:25:37.892556 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:37.892524    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6"]
Jul 28 12:25:37.896044 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:37.896022    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6"
Jul 28 12:25:37.900029 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:37.900007    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a4d289\""
Jul 28 12:25:37.900168 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:37.900008    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a4d289-dockercfg-9ckbg\""
Jul 28 12:25:37.907735 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:37.907711    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6"]
Jul 28 12:25:37.927247 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:37.927209    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg" event={"ID":"b39f2b10-84e5-433d-a9ee-c4e68f9f7650","Type":"ContainerStarted","Data":"d5df877636e1341aefc55ea64ff036b9f9fe8cb4097781259eed9514260de383"}
Jul 28 12:25:37.927412 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:37.927394    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg" event={"ID":"b39f2b10-84e5-433d-a9ee-c4e68f9f7650","Type":"ContainerStarted","Data":"dab7e73cdb83ab6cd296ac7526bd3133ff98f3bec0e14b2c709e391dcdab0506"}
Jul 28 12:25:37.927480 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:37.927451    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"
Jul 28 12:25:37.945758 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:37.945685    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg" podStartSLOduration=1.62804889 podStartE2EDuration="1.945672223s" totalImagesPullingTime="317.623333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:25:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:25:37.944237011 +0000 UTC m=+3289.802893033" watchObservedRunningTime="2026-07-28 12:25:37.945672223 +0000 UTC m=+3289.804328221"
Jul 28 12:25:37.953927 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:37.953902    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08294600-509c-4997-9eea-c4e1abeedb3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6\" (UID: \"08294600-509c-4997-9eea-c4e1abeedb3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6"
Jul 28 12:25:37.954038 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:37.953949    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5zvp\" (UniqueName: \"kubernetes.io/projected/08294600-509c-4997-9eea-c4e1abeedb3e-kube-api-access-g5zvp\") pod \"payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6\" (UID: \"08294600-509c-4997-9eea-c4e1abeedb3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6"
Jul 28 12:25:38.055429 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:38.055393    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08294600-509c-4997-9eea-c4e1abeedb3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6\" (UID: \"08294600-509c-4997-9eea-c4e1abeedb3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6"
Jul 28 12:25:38.055685 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:38.055663    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5zvp\" (UniqueName: \"kubernetes.io/projected/08294600-509c-4997-9eea-c4e1abeedb3e-kube-api-access-g5zvp\") pod \"payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6\" (UID: \"08294600-509c-4997-9eea-c4e1abeedb3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6"
Jul 28 12:25:38.056239 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:38.056217    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08294600-509c-4997-9eea-c4e1abeedb3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6\" (UID: \"08294600-509c-4997-9eea-c4e1abeedb3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6"
Jul 28 12:25:38.063476 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:38.063454    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5zvp\" (UniqueName: \"kubernetes.io/projected/08294600-509c-4997-9eea-c4e1abeedb3e-kube-api-access-g5zvp\") pod \"payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6\" (UID: \"08294600-509c-4997-9eea-c4e1abeedb3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6"
Jul 28 12:25:38.206753 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:38.206693    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6"
Jul 28 12:25:38.545549 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:25:38.545517    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08294600_509c_4997_9eea_c4e1abeedb3e.slice/crio-48ec1412873306e451f4d88019ac575573ab7882a549451e09156a11fa29dfa5 WatchSource:0}: Error finding container 48ec1412873306e451f4d88019ac575573ab7882a549451e09156a11fa29dfa5: Status 404 returned error can't find the container with id 48ec1412873306e451f4d88019ac575573ab7882a549451e09156a11fa29dfa5
Jul 28 12:25:38.547754 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:38.547730    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6"]
Jul 28 12:25:38.933013 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:38.932979    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6" event={"ID":"08294600-509c-4997-9eea-c4e1abeedb3e","Type":"ContainerStarted","Data":"2c45d679be4ee88bf37da5cbd56a05d6924a9d3497603e43c4a0ddac017a3327"}
Jul 28 12:25:38.933439 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:38.933040    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6"
Jul 28 12:25:38.933439 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:38.933058    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6" event={"ID":"08294600-509c-4997-9eea-c4e1abeedb3e","Type":"ContainerStarted","Data":"48ec1412873306e451f4d88019ac575573ab7882a549451e09156a11fa29dfa5"}
Jul 28 12:25:38.947061 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:38.947008    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6" podStartSLOduration=1.94699106 podStartE2EDuration="1.94699106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:25:38.946571524 +0000 UTC m=+3290.805227537" watchObservedRunningTime="2026-07-28 12:25:38.94699106 +0000 UTC m=+3290.805647061"
Jul 28 12:25:43.938847 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:43.938815    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"
Jul 28 12:25:44.329020 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:44.328982    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:25:49.939877 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:25:49.939846    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6"
Jul 28 12:26:05.207112 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.207072    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"]
Jul 28 12:26:05.210475 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.210451    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:05.212350 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.212329    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ba0125\"/\"default-dockercfg-jkflw\""
Jul 28 12:26:05.212982 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.212962    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ba0125\"/\"kube-root-ca.crt\""
Jul 28 12:26:05.213155 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.213137    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ba0125\"/\"openshift-service-ca.crt\""
Jul 28 12:26:05.213218 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.213154    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ba0125\"/\"ipp-route-ba0125-kserve-self-signed-certs\""
Jul 28 12:26:05.220119 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.220098    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"]
Jul 28 12:26:05.399679 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.399644    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-home\") pod \"ipp-route-ba0125-kserve-578d4c4bf9-zkvln\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") " pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:05.399833 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.399686    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-model-cache\") pod \"ipp-route-ba0125-kserve-578d4c4bf9-zkvln\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") " pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:05.399833 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.399708    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fljr5\" (UniqueName: \"kubernetes.io/projected/8a6624dc-ed2a-460c-a91d-dec3548ee355-kube-api-access-fljr5\") pod \"ipp-route-ba0125-kserve-578d4c4bf9-zkvln\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") " pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:05.399833 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.399799    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-dshm\") pod \"ipp-route-ba0125-kserve-578d4c4bf9-zkvln\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") " pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:05.399989 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.399835    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-tmp-dir\") pod \"ipp-route-ba0125-kserve-578d4c4bf9-zkvln\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") " pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:05.399989 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.399923    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6624dc-ed2a-460c-a91d-dec3548ee355-tls-certs\") pod \"ipp-route-ba0125-kserve-578d4c4bf9-zkvln\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") " pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:05.501028 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.500957    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-model-cache\") pod \"ipp-route-ba0125-kserve-578d4c4bf9-zkvln\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") " pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:05.501028 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.500988    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fljr5\" (UniqueName: \"kubernetes.io/projected/8a6624dc-ed2a-460c-a91d-dec3548ee355-kube-api-access-fljr5\") pod \"ipp-route-ba0125-kserve-578d4c4bf9-zkvln\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") " pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:05.501028 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.501012    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-dshm\") pod \"ipp-route-ba0125-kserve-578d4c4bf9-zkvln\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") " pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:05.501238 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.501208    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-tmp-dir\") pod \"ipp-route-ba0125-kserve-578d4c4bf9-zkvln\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") " pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:05.501315 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.501298    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6624dc-ed2a-460c-a91d-dec3548ee355-tls-certs\") pod \"ipp-route-ba0125-kserve-578d4c4bf9-zkvln\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") " pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:05.501404 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.501378    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-model-cache\") pod \"ipp-route-ba0125-kserve-578d4c4bf9-zkvln\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") " pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:05.501404 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.501380    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-home\") pod \"ipp-route-ba0125-kserve-578d4c4bf9-zkvln\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") " pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:05.501617 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.501568    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-tmp-dir\") pod \"ipp-route-ba0125-kserve-578d4c4bf9-zkvln\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") " pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:05.501617 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.501608    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-home\") pod \"ipp-route-ba0125-kserve-578d4c4bf9-zkvln\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") " pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:05.503242 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.503221    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-dshm\") pod \"ipp-route-ba0125-kserve-578d4c4bf9-zkvln\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") " pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:05.503470 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.503453    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6624dc-ed2a-460c-a91d-dec3548ee355-tls-certs\") pod \"ipp-route-ba0125-kserve-578d4c4bf9-zkvln\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") " pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:05.508291 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.508271    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fljr5\" (UniqueName: \"kubernetes.io/projected/8a6624dc-ed2a-460c-a91d-dec3548ee355-kube-api-access-fljr5\") pod \"ipp-route-ba0125-kserve-578d4c4bf9-zkvln\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") " pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:05.520308 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.520283    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:05.651127 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:05.651103    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"]
Jul 28 12:26:05.653167 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:26:05.653140    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a6624dc_ed2a_460c_a91d_dec3548ee355.slice/crio-b9c9f856022f0a1d42e9948fe4ebd69317a247d5d5d7a7003adb6d56f28e08fd WatchSource:0}: Error finding container b9c9f856022f0a1d42e9948fe4ebd69317a247d5d5d7a7003adb6d56f28e08fd: Status 404 returned error can't find the container with id b9c9f856022f0a1d42e9948fe4ebd69317a247d5d5d7a7003adb6d56f28e08fd
Jul 28 12:26:06.058822 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:06.058783    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln" event={"ID":"8a6624dc-ed2a-460c-a91d-dec3548ee355","Type":"ContainerStarted","Data":"d21225939e7628a987074c765fd65ff39747b800769d89c8794a3e282df9c0f9"}
Jul 28 12:26:06.058999 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:06.058837    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln" event={"ID":"8a6624dc-ed2a-460c-a91d-dec3548ee355","Type":"ContainerStarted","Data":"b9c9f856022f0a1d42e9948fe4ebd69317a247d5d5d7a7003adb6d56f28e08fd"}
Jul 28 12:26:08.839182 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:08.839130    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln" podStartSLOduration=3.839113136 podStartE2EDuration="3.839113136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:26:06.074196447 +0000 UTC m=+3317.932852445" watchObservedRunningTime="2026-07-28 12:26:08.839113136 +0000 UTC m=+3320.697769135"
Jul 28 12:26:08.839688 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:08.839629    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:26:15.520605 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:15.520476    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:15.520605 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:15.520544    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:15.533297 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:15.533267    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:16.124065 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:16.118404    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:26:39.429765 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:39.429729    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:26:49.127380 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:49.127351    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49_b8474023-b495-4710-9062-6e008ff5c712/payload-processing/0.log"
Jul 28 12:26:54.280090 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:54.280052    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp"]
Jul 28 12:26:54.283682 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:54.283665    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp"
Jul 28 12:26:54.285952 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:54.285925    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-0f3ba4\""
Jul 28 12:26:54.286084 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:54.285973    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-0f3ba4-dockercfg-8tjjx\""
Jul 28 12:26:54.292082 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:54.292056    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp"]
Jul 28 12:26:54.333801 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:54.333766    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plv9p\" (UniqueName: \"kubernetes.io/projected/f2e2a1a6-9ac3-4969-a0cf-57482a3964fe-kube-api-access-plv9p\") pod \"payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp\" (UID: \"f2e2a1a6-9ac3-4969-a0cf-57482a3964fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp"
Jul 28 12:26:54.333949 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:54.333813    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e2a1a6-9ac3-4969-a0cf-57482a3964fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp\" (UID: \"f2e2a1a6-9ac3-4969-a0cf-57482a3964fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp"
Jul 28 12:26:54.434789 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:54.434757    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plv9p\" (UniqueName: \"kubernetes.io/projected/f2e2a1a6-9ac3-4969-a0cf-57482a3964fe-kube-api-access-plv9p\") pod \"payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp\" (UID: \"f2e2a1a6-9ac3-4969-a0cf-57482a3964fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp"
Jul 28 12:26:54.434964 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:54.434810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e2a1a6-9ac3-4969-a0cf-57482a3964fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp\" (UID: \"f2e2a1a6-9ac3-4969-a0cf-57482a3964fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp"
Jul 28 12:26:54.435526 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:54.435503    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e2a1a6-9ac3-4969-a0cf-57482a3964fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp\" (UID: \"f2e2a1a6-9ac3-4969-a0cf-57482a3964fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp"
Jul 28 12:26:54.442522 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:54.442500    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plv9p\" (UniqueName: \"kubernetes.io/projected/f2e2a1a6-9ac3-4969-a0cf-57482a3964fe-kube-api-access-plv9p\") pod \"payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp\" (UID: \"f2e2a1a6-9ac3-4969-a0cf-57482a3964fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp"
Jul 28 12:26:54.596622 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:54.596593    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp"
Jul 28 12:26:54.725493 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:54.725469    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp"]
Jul 28 12:26:54.728196 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:26:54.728165    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e2a1a6_9ac3_4969_a0cf_57482a3964fe.slice/crio-730c62dcdd2525026556ee620ea2e2f577846226c5b5c9598e90c1414a46f431 WatchSource:0}: Error finding container 730c62dcdd2525026556ee620ea2e2f577846226c5b5c9598e90c1414a46f431: Status 404 returned error can't find the container with id 730c62dcdd2525026556ee620ea2e2f577846226c5b5c9598e90c1414a46f431
Jul 28 12:26:55.303329 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:55.303286    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp" event={"ID":"f2e2a1a6-9ac3-4969-a0cf-57482a3964fe","Type":"ContainerStarted","Data":"f5aa0434f539ac3ec83e2b6d2552278b96b36eaafab8f2209443a9ac819f859d"}
Jul 28 12:26:55.303750 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:55.303373    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp" event={"ID":"f2e2a1a6-9ac3-4969-a0cf-57482a3964fe","Type":"ContainerStarted","Data":"730c62dcdd2525026556ee620ea2e2f577846226c5b5c9598e90c1414a46f431"}
Jul 28 12:26:55.303750 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:55.303414    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp"
Jul 28 12:26:55.325210 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:55.325152    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp" podStartSLOduration=1.32513761 podStartE2EDuration="1.32513761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:26:55.323375042 +0000 UTC m=+3367.182031055" watchObservedRunningTime="2026-07-28 12:26:55.32513761 +0000 UTC m=+3367.183793610"
Jul 28 12:26:58.323196 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:26:58.323164    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:27:06.309904 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:06.309877    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp"
Jul 28 12:27:19.320129 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:19.320085    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp"]
Jul 28 12:27:19.320634 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:19.320338    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp" podUID="f2e2a1a6-9ac3-4969-a0cf-57482a3964fe" containerName="payload-pre-processing" containerID="cri-o://f5aa0434f539ac3ec83e2b6d2552278b96b36eaafab8f2209443a9ac819f859d" gracePeriod=30
Jul 28 12:27:19.567471 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:19.567443    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp"
Jul 28 12:27:19.631012 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:19.630934    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plv9p\" (UniqueName: \"kubernetes.io/projected/f2e2a1a6-9ac3-4969-a0cf-57482a3964fe-kube-api-access-plv9p\") pod \"f2e2a1a6-9ac3-4969-a0cf-57482a3964fe\" (UID: \"f2e2a1a6-9ac3-4969-a0cf-57482a3964fe\") "
Jul 28 12:27:19.631149 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:19.631023    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e2a1a6-9ac3-4969-a0cf-57482a3964fe-plugins-config-volume\") pod \"f2e2a1a6-9ac3-4969-a0cf-57482a3964fe\" (UID: \"f2e2a1a6-9ac3-4969-a0cf-57482a3964fe\") "
Jul 28 12:27:19.631378 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:19.631348    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e2a1a6-9ac3-4969-a0cf-57482a3964fe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2e2a1a6-9ac3-4969-a0cf-57482a3964fe" (UID: "f2e2a1a6-9ac3-4969-a0cf-57482a3964fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:27:19.633093 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:19.633061    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e2a1a6-9ac3-4969-a0cf-57482a3964fe-kube-api-access-plv9p" (OuterVolumeSpecName: "kube-api-access-plv9p") pod "f2e2a1a6-9ac3-4969-a0cf-57482a3964fe" (UID: "f2e2a1a6-9ac3-4969-a0cf-57482a3964fe"). InnerVolumeSpecName "kube-api-access-plv9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:27:19.732335 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:19.732306    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plv9p\" (UniqueName: \"kubernetes.io/projected/f2e2a1a6-9ac3-4969-a0cf-57482a3964fe-kube-api-access-plv9p\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:27:19.732335 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:19.732332    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e2a1a6-9ac3-4969-a0cf-57482a3964fe-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:27:20.419447 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:20.419414    2561 generic.go:356] "Generic (PLEG): container finished" podID="f2e2a1a6-9ac3-4969-a0cf-57482a3964fe" containerID="f5aa0434f539ac3ec83e2b6d2552278b96b36eaafab8f2209443a9ac819f859d" exitCode=0
Jul 28 12:27:20.419955 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:20.419469    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp"
Jul 28 12:27:20.419955 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:20.419463    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp" event={"ID":"f2e2a1a6-9ac3-4969-a0cf-57482a3964fe","Type":"ContainerDied","Data":"f5aa0434f539ac3ec83e2b6d2552278b96b36eaafab8f2209443a9ac819f859d"}
Jul 28 12:27:20.419955 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:20.419623    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp" event={"ID":"f2e2a1a6-9ac3-4969-a0cf-57482a3964fe","Type":"ContainerDied","Data":"730c62dcdd2525026556ee620ea2e2f577846226c5b5c9598e90c1414a46f431"}
Jul 28 12:27:20.419955 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:20.419648    2561 scope.go:117] "RemoveContainer" containerID="f5aa0434f539ac3ec83e2b6d2552278b96b36eaafab8f2209443a9ac819f859d"
Jul 28 12:27:20.428367 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:20.428349    2561 scope.go:117] "RemoveContainer" containerID="f5aa0434f539ac3ec83e2b6d2552278b96b36eaafab8f2209443a9ac819f859d"
Jul 28 12:27:20.428662 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:27:20.428638    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5aa0434f539ac3ec83e2b6d2552278b96b36eaafab8f2209443a9ac819f859d\": container with ID starting with f5aa0434f539ac3ec83e2b6d2552278b96b36eaafab8f2209443a9ac819f859d not found: ID does not exist" containerID="f5aa0434f539ac3ec83e2b6d2552278b96b36eaafab8f2209443a9ac819f859d"
Jul 28 12:27:20.428762 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:20.428668    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5aa0434f539ac3ec83e2b6d2552278b96b36eaafab8f2209443a9ac819f859d"} err="failed to get container status \"f5aa0434f539ac3ec83e2b6d2552278b96b36eaafab8f2209443a9ac819f859d\": rpc error: code = NotFound desc = could not find container \"f5aa0434f539ac3ec83e2b6d2552278b96b36eaafab8f2209443a9ac819f859d\": container with ID starting with f5aa0434f539ac3ec83e2b6d2552278b96b36eaafab8f2209443a9ac819f859d not found: ID does not exist"
Jul 28 12:27:20.441058 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:20.441039    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp"]
Jul 28 12:27:20.444768 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:20.444747    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-0f3ba4-898dc6cd8-2fjfp"]
Jul 28 12:27:20.547171 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:20.547140    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2e2a1a6-9ac3-4969-a0cf-57482a3964fe" path="/var/lib/kubelet/pods/f2e2a1a6-9ac3-4969-a0cf-57482a3964fe/volumes"
Jul 28 12:27:23.928180 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:23.928141    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:27:49.631762 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:49.631725    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:27:50.897478 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:50.897447    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49"]
Jul 28 12:27:50.897921 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:50.897700    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49" podUID="b8474023-b495-4710-9062-6e008ff5c712" containerName="payload-processing" containerID="cri-o://aeaf406ecf0c1808e971df51ed959ae7b32751894f9a44bbfeb9922149db12fd" gracePeriod=30
Jul 28 12:27:50.904997 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:50.904969    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv"]
Jul 28 12:27:50.905250 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:50.905203    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv" podUID="cbd0f8c3-2b81-4319-b624-5104697eb0ee" containerName="payload-pre-processing" containerID="cri-o://de1a65ea8eebfebfa4eafbc401e31e2c96d9a2de0952538cb0d8e0643c2b1f0d" gracePeriod=30
Jul 28 12:27:51.185093 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.185072    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49"
Jul 28 12:27:51.187738 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.187718    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv"
Jul 28 12:27:51.317932 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.317905    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqhnh\" (UniqueName: \"kubernetes.io/projected/b8474023-b495-4710-9062-6e008ff5c712-kube-api-access-bqhnh\") pod \"b8474023-b495-4710-9062-6e008ff5c712\" (UID: \"b8474023-b495-4710-9062-6e008ff5c712\") "
Jul 28 12:27:51.318107 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.317963    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwtg5\" (UniqueName: \"kubernetes.io/projected/cbd0f8c3-2b81-4319-b624-5104697eb0ee-kube-api-access-zwtg5\") pod \"cbd0f8c3-2b81-4319-b624-5104697eb0ee\" (UID: \"cbd0f8c3-2b81-4319-b624-5104697eb0ee\") "
Jul 28 12:27:51.318107 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.318031    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8474023-b495-4710-9062-6e008ff5c712-plugins-config-volume\") pod \"b8474023-b495-4710-9062-6e008ff5c712\" (UID: \"b8474023-b495-4710-9062-6e008ff5c712\") "
Jul 28 12:27:51.318107 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.318068    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbd0f8c3-2b81-4319-b624-5104697eb0ee-plugins-config-volume\") pod \"cbd0f8c3-2b81-4319-b624-5104697eb0ee\" (UID: \"cbd0f8c3-2b81-4319-b624-5104697eb0ee\") "
Jul 28 12:27:51.318425 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.318397    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8474023-b495-4710-9062-6e008ff5c712-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b8474023-b495-4710-9062-6e008ff5c712" (UID: "b8474023-b495-4710-9062-6e008ff5c712"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:27:51.318510 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.318470    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbd0f8c3-2b81-4319-b624-5104697eb0ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cbd0f8c3-2b81-4319-b624-5104697eb0ee" (UID: "cbd0f8c3-2b81-4319-b624-5104697eb0ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:27:51.320054 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.320030    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd0f8c3-2b81-4319-b624-5104697eb0ee-kube-api-access-zwtg5" (OuterVolumeSpecName: "kube-api-access-zwtg5") pod "cbd0f8c3-2b81-4319-b624-5104697eb0ee" (UID: "cbd0f8c3-2b81-4319-b624-5104697eb0ee"). InnerVolumeSpecName "kube-api-access-zwtg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:27:51.320168 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.320118    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8474023-b495-4710-9062-6e008ff5c712-kube-api-access-bqhnh" (OuterVolumeSpecName: "kube-api-access-bqhnh") pod "b8474023-b495-4710-9062-6e008ff5c712" (UID: "b8474023-b495-4710-9062-6e008ff5c712"). InnerVolumeSpecName "kube-api-access-bqhnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:27:51.419685 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.419618    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwtg5\" (UniqueName: \"kubernetes.io/projected/cbd0f8c3-2b81-4319-b624-5104697eb0ee-kube-api-access-zwtg5\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:27:51.419685 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.419650    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8474023-b495-4710-9062-6e008ff5c712-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:27:51.419685 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.419659    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbd0f8c3-2b81-4319-b624-5104697eb0ee-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:27:51.419685 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.419668    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqhnh\" (UniqueName: \"kubernetes.io/projected/b8474023-b495-4710-9062-6e008ff5c712-kube-api-access-bqhnh\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:27:51.570575 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.570543    2561 generic.go:356] "Generic (PLEG): container finished" podID="b8474023-b495-4710-9062-6e008ff5c712" containerID="aeaf406ecf0c1808e971df51ed959ae7b32751894f9a44bbfeb9922149db12fd" exitCode=0
Jul 28 12:27:51.570575 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.570594    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49"
Jul 28 12:27:51.570785 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.570629    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49" event={"ID":"b8474023-b495-4710-9062-6e008ff5c712","Type":"ContainerDied","Data":"aeaf406ecf0c1808e971df51ed959ae7b32751894f9a44bbfeb9922149db12fd"}
Jul 28 12:27:51.570785 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.570681    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49" event={"ID":"b8474023-b495-4710-9062-6e008ff5c712","Type":"ContainerDied","Data":"147c31432b94f6b15b7d3ebb8bfa68f53ffbe05add546fd2b74343beaf9a4f96"}
Jul 28 12:27:51.570785 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.570710    2561 scope.go:117] "RemoveContainer" containerID="aeaf406ecf0c1808e971df51ed959ae7b32751894f9a44bbfeb9922149db12fd"
Jul 28 12:27:51.572134 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.572108    2561 generic.go:356] "Generic (PLEG): container finished" podID="cbd0f8c3-2b81-4319-b624-5104697eb0ee" containerID="de1a65ea8eebfebfa4eafbc401e31e2c96d9a2de0952538cb0d8e0643c2b1f0d" exitCode=0
Jul 28 12:27:51.572253 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.572149    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv"
Jul 28 12:27:51.572253 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.572189    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv" event={"ID":"cbd0f8c3-2b81-4319-b624-5104697eb0ee","Type":"ContainerDied","Data":"de1a65ea8eebfebfa4eafbc401e31e2c96d9a2de0952538cb0d8e0643c2b1f0d"}
Jul 28 12:27:51.572253 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.572230    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv" event={"ID":"cbd0f8c3-2b81-4319-b624-5104697eb0ee","Type":"ContainerDied","Data":"89a01ba98a5748ac6e03f5422c21f1167afb96fb867462958de58f3f8ac40f3e"}
Jul 28 12:27:51.580882 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.580864    2561 scope.go:117] "RemoveContainer" containerID="aeaf406ecf0c1808e971df51ed959ae7b32751894f9a44bbfeb9922149db12fd"
Jul 28 12:27:51.581158 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:27:51.581131    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeaf406ecf0c1808e971df51ed959ae7b32751894f9a44bbfeb9922149db12fd\": container with ID starting with aeaf406ecf0c1808e971df51ed959ae7b32751894f9a44bbfeb9922149db12fd not found: ID does not exist" containerID="aeaf406ecf0c1808e971df51ed959ae7b32751894f9a44bbfeb9922149db12fd"
Jul 28 12:27:51.581226 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.581172    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeaf406ecf0c1808e971df51ed959ae7b32751894f9a44bbfeb9922149db12fd"} err="failed to get container status \"aeaf406ecf0c1808e971df51ed959ae7b32751894f9a44bbfeb9922149db12fd\": rpc error: code = NotFound desc = could not find container \"aeaf406ecf0c1808e971df51ed959ae7b32751894f9a44bbfeb9922149db12fd\": container with ID starting with aeaf406ecf0c1808e971df51ed959ae7b32751894f9a44bbfeb9922149db12fd not found: ID does not exist"
Jul 28 12:27:51.581226 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.581205    2561 scope.go:117] "RemoveContainer" containerID="de1a65ea8eebfebfa4eafbc401e31e2c96d9a2de0952538cb0d8e0643c2b1f0d"
Jul 28 12:27:51.588718 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.588702    2561 scope.go:117] "RemoveContainer" containerID="de1a65ea8eebfebfa4eafbc401e31e2c96d9a2de0952538cb0d8e0643c2b1f0d"
Jul 28 12:27:51.588940 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:27:51.588924    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1a65ea8eebfebfa4eafbc401e31e2c96d9a2de0952538cb0d8e0643c2b1f0d\": container with ID starting with de1a65ea8eebfebfa4eafbc401e31e2c96d9a2de0952538cb0d8e0643c2b1f0d not found: ID does not exist" containerID="de1a65ea8eebfebfa4eafbc401e31e2c96d9a2de0952538cb0d8e0643c2b1f0d"
Jul 28 12:27:51.589000 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.588944    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1a65ea8eebfebfa4eafbc401e31e2c96d9a2de0952538cb0d8e0643c2b1f0d"} err="failed to get container status \"de1a65ea8eebfebfa4eafbc401e31e2c96d9a2de0952538cb0d8e0643c2b1f0d\": rpc error: code = NotFound desc = could not find container \"de1a65ea8eebfebfa4eafbc401e31e2c96d9a2de0952538cb0d8e0643c2b1f0d\": container with ID starting with de1a65ea8eebfebfa4eafbc401e31e2c96d9a2de0952538cb0d8e0643c2b1f0d not found: ID does not exist"
Jul 28 12:27:51.593032 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.593010    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49"]
Jul 28 12:27:51.597294 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.597265    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba0125-78c9c97d76-5vv49"]
Jul 28 12:27:51.608558 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.608528    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv"]
Jul 28 12:27:51.612475 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:51.612457    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba0125-7686c7f764-5pqnv"]
Jul 28 12:27:52.546627 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:52.546571    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8474023-b495-4710-9062-6e008ff5c712" path="/var/lib/kubelet/pods/b8474023-b495-4710-9062-6e008ff5c712/volumes"
Jul 28 12:27:52.547116 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:27:52.547089    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbd0f8c3-2b81-4319-b624-5104697eb0ee" path="/var/lib/kubelet/pods/cbd0f8c3-2b81-4319-b624-5104697eb0ee/volumes"
Jul 28 12:28:00.781460 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:00.781424    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"]
Jul 28 12:28:02.323572 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:02.323530    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:28:15.800407 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:15.800346    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln" podUID="8a6624dc-ed2a-460c-a91d-dec3548ee355" containerName="main" containerID="cri-o://d21225939e7628a987074c765fd65ff39747b800769d89c8794a3e282df9c0f9" gracePeriod=45
Jul 28 12:28:16.057485 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.057461    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:28:16.119308 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.119279    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-home\") pod \"8a6624dc-ed2a-460c-a91d-dec3548ee355\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") "
Jul 28 12:28:16.119485 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.119345    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-dshm\") pod \"8a6624dc-ed2a-460c-a91d-dec3548ee355\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") "
Jul 28 12:28:16.119485 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.119379    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-model-cache\") pod \"8a6624dc-ed2a-460c-a91d-dec3548ee355\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") "
Jul 28 12:28:16.119485 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.119443    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fljr5\" (UniqueName: \"kubernetes.io/projected/8a6624dc-ed2a-460c-a91d-dec3548ee355-kube-api-access-fljr5\") pod \"8a6624dc-ed2a-460c-a91d-dec3548ee355\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") "
Jul 28 12:28:16.119485 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.119473    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-tmp-dir\") pod \"8a6624dc-ed2a-460c-a91d-dec3548ee355\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") "
Jul 28 12:28:16.119701 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.119520    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6624dc-ed2a-460c-a91d-dec3548ee355-tls-certs\") pod \"8a6624dc-ed2a-460c-a91d-dec3548ee355\" (UID: \"8a6624dc-ed2a-460c-a91d-dec3548ee355\") "
Jul 28 12:28:16.119701 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.119626    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-home" (OuterVolumeSpecName: "home") pod "8a6624dc-ed2a-460c-a91d-dec3548ee355" (UID: "8a6624dc-ed2a-460c-a91d-dec3548ee355"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:28:16.119821 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.119746    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a6624dc-ed2a-460c-a91d-dec3548ee355" (UID: "8a6624dc-ed2a-460c-a91d-dec3548ee355"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:28:16.119875 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.119838    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-home\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:28:16.119875 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.119851    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-tmp-dir\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:28:16.119981 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.119948    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a6624dc-ed2a-460c-a91d-dec3548ee355" (UID: "8a6624dc-ed2a-460c-a91d-dec3548ee355"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:28:16.121737 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.121706    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6624dc-ed2a-460c-a91d-dec3548ee355-kube-api-access-fljr5" (OuterVolumeSpecName: "kube-api-access-fljr5") pod "8a6624dc-ed2a-460c-a91d-dec3548ee355" (UID: "8a6624dc-ed2a-460c-a91d-dec3548ee355"). InnerVolumeSpecName "kube-api-access-fljr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:28:16.121737 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.121710    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-dshm" (OuterVolumeSpecName: "dshm") pod "8a6624dc-ed2a-460c-a91d-dec3548ee355" (UID: "8a6624dc-ed2a-460c-a91d-dec3548ee355"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:28:16.121890 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.121753    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a6624dc-ed2a-460c-a91d-dec3548ee355-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a6624dc-ed2a-460c-a91d-dec3548ee355" (UID: "8a6624dc-ed2a-460c-a91d-dec3548ee355"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:28:16.220198 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.220167    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-dshm\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:28:16.220198 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.220187    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6624dc-ed2a-460c-a91d-dec3548ee355-model-cache\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:28:16.220198 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.220197    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fljr5\" (UniqueName: \"kubernetes.io/projected/8a6624dc-ed2a-460c-a91d-dec3548ee355-kube-api-access-fljr5\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:28:16.220453 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.220207    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6624dc-ed2a-460c-a91d-dec3548ee355-tls-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:28:16.691784 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.691750    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a6624dc-ed2a-460c-a91d-dec3548ee355" containerID="d21225939e7628a987074c765fd65ff39747b800769d89c8794a3e282df9c0f9" exitCode=0
Jul 28 12:28:16.691949 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.691795    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"
Jul 28 12:28:16.691949 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.691808    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln" event={"ID":"8a6624dc-ed2a-460c-a91d-dec3548ee355","Type":"ContainerDied","Data":"d21225939e7628a987074c765fd65ff39747b800769d89c8794a3e282df9c0f9"}
Jul 28 12:28:16.691949 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.691845    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln" event={"ID":"8a6624dc-ed2a-460c-a91d-dec3548ee355","Type":"ContainerDied","Data":"b9c9f856022f0a1d42e9948fe4ebd69317a247d5d5d7a7003adb6d56f28e08fd"}
Jul 28 12:28:16.691949 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.691868    2561 scope.go:117] "RemoveContainer" containerID="d21225939e7628a987074c765fd65ff39747b800769d89c8794a3e282df9c0f9"
Jul 28 12:28:16.702844 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.702829    2561 scope.go:117] "RemoveContainer" containerID="d21225939e7628a987074c765fd65ff39747b800769d89c8794a3e282df9c0f9"
Jul 28 12:28:16.703097 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:28:16.703079    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d21225939e7628a987074c765fd65ff39747b800769d89c8794a3e282df9c0f9\": container with ID starting with d21225939e7628a987074c765fd65ff39747b800769d89c8794a3e282df9c0f9 not found: ID does not exist" containerID="d21225939e7628a987074c765fd65ff39747b800769d89c8794a3e282df9c0f9"
Jul 28 12:28:16.703209 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.703104    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21225939e7628a987074c765fd65ff39747b800769d89c8794a3e282df9c0f9"} err="failed to get container status \"d21225939e7628a987074c765fd65ff39747b800769d89c8794a3e282df9c0f9\": rpc error: code = NotFound desc = could not find container \"d21225939e7628a987074c765fd65ff39747b800769d89c8794a3e282df9c0f9\": container with ID starting with d21225939e7628a987074c765fd65ff39747b800769d89c8794a3e282df9c0f9 not found: ID does not exist"
Jul 28 12:28:16.706105 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.706082    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"]
Jul 28 12:28:16.708799 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:16.708778    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-ba0125/ipp-route-ba0125-kserve-578d4c4bf9-zkvln"]
Jul 28 12:28:18.546327 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:18.546291    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a6624dc-ed2a-460c-a91d-dec3548ee355" path="/var/lib/kubelet/pods/8a6624dc-ed2a-460c-a91d-dec3548ee355/volumes"
Jul 28 12:28:56.865462 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:56.865424    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"]
Jul 28 12:28:56.866003 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:56.865669    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg" podUID="b39f2b10-84e5-433d-a9ee-c4e68f9f7650" containerName="maas-api" containerID="cri-o://d5df877636e1341aefc55ea64ff036b9f9fe8cb4097781259eed9514260de383" gracePeriod=30
Jul 28 12:28:56.990245 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:56.990209    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6"]
Jul 28 12:28:56.990494 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:56.990467    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6" podUID="08294600-509c-4997-9eea-c4e1abeedb3e" containerName="payload-pre-processing" containerID="cri-o://2c45d679be4ee88bf37da5cbd56a05d6924a9d3497603e43c4a0ddac017a3327" gracePeriod=30
Jul 28 12:28:57.121810 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.121749    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"
Jul 28 12:28:57.169704 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.169670    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhlp5\" (UniqueName: \"kubernetes.io/projected/b39f2b10-84e5-433d-a9ee-c4e68f9f7650-kube-api-access-fhlp5\") pod \"b39f2b10-84e5-433d-a9ee-c4e68f9f7650\" (UID: \"b39f2b10-84e5-433d-a9ee-c4e68f9f7650\") "
Jul 28 12:28:57.169888 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.169871    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b39f2b10-84e5-433d-a9ee-c4e68f9f7650-maas-api-tls\") pod \"b39f2b10-84e5-433d-a9ee-c4e68f9f7650\" (UID: \"b39f2b10-84e5-433d-a9ee-c4e68f9f7650\") "
Jul 28 12:28:57.169962 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.169931    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b39f2b10-84e5-433d-a9ee-c4e68f9f7650-openshift-service-ca\") pod \"b39f2b10-84e5-433d-a9ee-c4e68f9f7650\" (UID: \"b39f2b10-84e5-433d-a9ee-c4e68f9f7650\") "
Jul 28 12:28:57.170346 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.170315    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39f2b10-84e5-433d-a9ee-c4e68f9f7650-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b39f2b10-84e5-433d-a9ee-c4e68f9f7650" (UID: "b39f2b10-84e5-433d-a9ee-c4e68f9f7650"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:28:57.172634 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.172605    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39f2b10-84e5-433d-a9ee-c4e68f9f7650-kube-api-access-fhlp5" (OuterVolumeSpecName: "kube-api-access-fhlp5") pod "b39f2b10-84e5-433d-a9ee-c4e68f9f7650" (UID: "b39f2b10-84e5-433d-a9ee-c4e68f9f7650"). InnerVolumeSpecName "kube-api-access-fhlp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:28:57.172634 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.172623    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b39f2b10-84e5-433d-a9ee-c4e68f9f7650-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b39f2b10-84e5-433d-a9ee-c4e68f9f7650" (UID: "b39f2b10-84e5-433d-a9ee-c4e68f9f7650"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:28:57.250303 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.250277    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6"
Jul 28 12:28:57.271341 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.271319    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5zvp\" (UniqueName: \"kubernetes.io/projected/08294600-509c-4997-9eea-c4e1abeedb3e-kube-api-access-g5zvp\") pod \"08294600-509c-4997-9eea-c4e1abeedb3e\" (UID: \"08294600-509c-4997-9eea-c4e1abeedb3e\") "
Jul 28 12:28:57.271471 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.271402    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08294600-509c-4997-9eea-c4e1abeedb3e-plugins-config-volume\") pod \"08294600-509c-4997-9eea-c4e1abeedb3e\" (UID: \"08294600-509c-4997-9eea-c4e1abeedb3e\") "
Jul 28 12:28:57.271561 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.271546    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhlp5\" (UniqueName: \"kubernetes.io/projected/b39f2b10-84e5-433d-a9ee-c4e68f9f7650-kube-api-access-fhlp5\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:28:57.271641 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.271561    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b39f2b10-84e5-433d-a9ee-c4e68f9f7650-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:28:57.271641 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.271572    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b39f2b10-84e5-433d-a9ee-c4e68f9f7650-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:28:57.271822 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.271801    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08294600-509c-4997-9eea-c4e1abeedb3e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "08294600-509c-4997-9eea-c4e1abeedb3e" (UID: "08294600-509c-4997-9eea-c4e1abeedb3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:28:57.273398 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.273375    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08294600-509c-4997-9eea-c4e1abeedb3e-kube-api-access-g5zvp" (OuterVolumeSpecName: "kube-api-access-g5zvp") pod "08294600-509c-4997-9eea-c4e1abeedb3e" (UID: "08294600-509c-4997-9eea-c4e1abeedb3e"). InnerVolumeSpecName "kube-api-access-g5zvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:28:57.373066 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.372981    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5zvp\" (UniqueName: \"kubernetes.io/projected/08294600-509c-4997-9eea-c4e1abeedb3e-kube-api-access-g5zvp\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:28:57.373066 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.373017    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08294600-509c-4997-9eea-c4e1abeedb3e-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:28:57.873516 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.873481    2561 generic.go:356] "Generic (PLEG): container finished" podID="08294600-509c-4997-9eea-c4e1abeedb3e" containerID="2c45d679be4ee88bf37da5cbd56a05d6924a9d3497603e43c4a0ddac017a3327" exitCode=0
Jul 28 12:28:57.873963 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.873531    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6"
Jul 28 12:28:57.873963 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.873562    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6" event={"ID":"08294600-509c-4997-9eea-c4e1abeedb3e","Type":"ContainerDied","Data":"2c45d679be4ee88bf37da5cbd56a05d6924a9d3497603e43c4a0ddac017a3327"}
Jul 28 12:28:57.873963 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.873636    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6" event={"ID":"08294600-509c-4997-9eea-c4e1abeedb3e","Type":"ContainerDied","Data":"48ec1412873306e451f4d88019ac575573ab7882a549451e09156a11fa29dfa5"}
Jul 28 12:28:57.873963 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.873669    2561 scope.go:117] "RemoveContainer" containerID="2c45d679be4ee88bf37da5cbd56a05d6924a9d3497603e43c4a0ddac017a3327"
Jul 28 12:28:57.875263 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.875090    2561 generic.go:356] "Generic (PLEG): container finished" podID="b39f2b10-84e5-433d-a9ee-c4e68f9f7650" containerID="d5df877636e1341aefc55ea64ff036b9f9fe8cb4097781259eed9514260de383" exitCode=0
Jul 28 12:28:57.875263 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.875129    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg" event={"ID":"b39f2b10-84e5-433d-a9ee-c4e68f9f7650","Type":"ContainerDied","Data":"d5df877636e1341aefc55ea64ff036b9f9fe8cb4097781259eed9514260de383"}
Jul 28 12:28:57.875263 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.875161    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"
Jul 28 12:28:57.875263 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.875164    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg" event={"ID":"b39f2b10-84e5-433d-a9ee-c4e68f9f7650","Type":"ContainerDied","Data":"dab7e73cdb83ab6cd296ac7526bd3133ff98f3bec0e14b2c709e391dcdab0506"}
Jul 28 12:28:57.890613 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.890595    2561 scope.go:117] "RemoveContainer" containerID="2c45d679be4ee88bf37da5cbd56a05d6924a9d3497603e43c4a0ddac017a3327"
Jul 28 12:28:57.890870 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:28:57.890853    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c45d679be4ee88bf37da5cbd56a05d6924a9d3497603e43c4a0ddac017a3327\": container with ID starting with 2c45d679be4ee88bf37da5cbd56a05d6924a9d3497603e43c4a0ddac017a3327 not found: ID does not exist" containerID="2c45d679be4ee88bf37da5cbd56a05d6924a9d3497603e43c4a0ddac017a3327"
Jul 28 12:28:57.890921 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.890879    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c45d679be4ee88bf37da5cbd56a05d6924a9d3497603e43c4a0ddac017a3327"} err="failed to get container status \"2c45d679be4ee88bf37da5cbd56a05d6924a9d3497603e43c4a0ddac017a3327\": rpc error: code = NotFound desc = could not find container \"2c45d679be4ee88bf37da5cbd56a05d6924a9d3497603e43c4a0ddac017a3327\": container with ID starting with 2c45d679be4ee88bf37da5cbd56a05d6924a9d3497603e43c4a0ddac017a3327 not found: ID does not exist"
Jul 28 12:28:57.890921 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.890896    2561 scope.go:117] "RemoveContainer" containerID="d5df877636e1341aefc55ea64ff036b9f9fe8cb4097781259eed9514260de383"
Jul 28 12:28:57.898393 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.898378    2561 scope.go:117] "RemoveContainer" containerID="d5df877636e1341aefc55ea64ff036b9f9fe8cb4097781259eed9514260de383"
Jul 28 12:28:57.898727 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:28:57.898708    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5df877636e1341aefc55ea64ff036b9f9fe8cb4097781259eed9514260de383\": container with ID starting with d5df877636e1341aefc55ea64ff036b9f9fe8cb4097781259eed9514260de383 not found: ID does not exist" containerID="d5df877636e1341aefc55ea64ff036b9f9fe8cb4097781259eed9514260de383"
Jul 28 12:28:57.898832 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.898732    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5df877636e1341aefc55ea64ff036b9f9fe8cb4097781259eed9514260de383"} err="failed to get container status \"d5df877636e1341aefc55ea64ff036b9f9fe8cb4097781259eed9514260de383\": rpc error: code = NotFound desc = could not find container \"d5df877636e1341aefc55ea64ff036b9f9fe8cb4097781259eed9514260de383\": container with ID starting with d5df877636e1341aefc55ea64ff036b9f9fe8cb4097781259eed9514260de383 not found: ID does not exist"
Jul 28 12:28:57.908373 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.908348    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"]
Jul 28 12:28:57.913020 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.912999    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a4d289-69cd4b994-nvdqg"]
Jul 28 12:28:57.936549 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.936528    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6"]
Jul 28 12:28:57.945914 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:57.945893    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a4d289-565c6576fb-swfh6"]
Jul 28 12:28:58.546325 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:58.546284    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08294600-509c-4997-9eea-c4e1abeedb3e" path="/var/lib/kubelet/pods/08294600-509c-4997-9eea-c4e1abeedb3e/volumes"
Jul 28 12:28:58.546773 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:28:58.546752    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b39f2b10-84e5-433d-a9ee-c4e68f9f7650" path="/var/lib/kubelet/pods/b39f2b10-84e5-433d-a9ee-c4e68f9f7650/volumes"
Jul 28 12:29:00.248919 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:00.248882    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:29:19.226645 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.226557    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"]
Jul 28 12:29:19.227153 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.226762    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx" podUID="830555b4-08e7-4df2-bc3f-2bbc3141c76e" containerName="maas-api" containerID="cri-o://f3cc8f81402bc7aa3e46b2ab27c91dcdc19b201a9848758a936cd68beaabc921" gracePeriod=30
Jul 28 12:29:19.323203 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.323169    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54"]
Jul 28 12:29:19.323437 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.323409    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54" podUID="35cb4004-f092-4b19-ae7c-5b582bb7f555" containerName="payload-processing" containerID="cri-o://b8083a2b6f4669b9c0cb8a58d26c92b8db2e428201acf1676491e526b1ffc461" gracePeriod=30
Jul 28 12:29:19.332657 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.331919    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6"]
Jul 28 12:29:19.332657 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.332150    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6" podUID="58592270-db06-499b-ab98-119e6331b69a" containerName="payload-pre-processing" containerID="cri-o://60999e73ba0b3a29abacb3472dc38cbc91f05b79c370d77e62670aa245c4fdb4" gracePeriod=30
Jul 28 12:29:19.486673 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.486608    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"
Jul 28 12:29:19.558223 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.558196    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/830555b4-08e7-4df2-bc3f-2bbc3141c76e-openshift-service-ca\") pod \"830555b4-08e7-4df2-bc3f-2bbc3141c76e\" (UID: \"830555b4-08e7-4df2-bc3f-2bbc3141c76e\") "
Jul 28 12:29:19.558359 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.558261    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqr47\" (UniqueName: \"kubernetes.io/projected/830555b4-08e7-4df2-bc3f-2bbc3141c76e-kube-api-access-zqr47\") pod \"830555b4-08e7-4df2-bc3f-2bbc3141c76e\" (UID: \"830555b4-08e7-4df2-bc3f-2bbc3141c76e\") "
Jul 28 12:29:19.558359 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.558299    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/830555b4-08e7-4df2-bc3f-2bbc3141c76e-maas-api-tls\") pod \"830555b4-08e7-4df2-bc3f-2bbc3141c76e\" (UID: \"830555b4-08e7-4df2-bc3f-2bbc3141c76e\") "
Jul 28 12:29:19.558622 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.558570    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830555b4-08e7-4df2-bc3f-2bbc3141c76e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "830555b4-08e7-4df2-bc3f-2bbc3141c76e" (UID: "830555b4-08e7-4df2-bc3f-2bbc3141c76e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:29:19.560333 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.560301    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/830555b4-08e7-4df2-bc3f-2bbc3141c76e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "830555b4-08e7-4df2-bc3f-2bbc3141c76e" (UID: "830555b4-08e7-4df2-bc3f-2bbc3141c76e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:29:19.560414 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.560369    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830555b4-08e7-4df2-bc3f-2bbc3141c76e-kube-api-access-zqr47" (OuterVolumeSpecName: "kube-api-access-zqr47") pod "830555b4-08e7-4df2-bc3f-2bbc3141c76e" (UID: "830555b4-08e7-4df2-bc3f-2bbc3141c76e"). InnerVolumeSpecName "kube-api-access-zqr47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:29:19.602992 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.602969    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54"
Jul 28 12:29:19.613053 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.613033    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6"
Jul 28 12:29:19.659303 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.659277    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/830555b4-08e7-4df2-bc3f-2bbc3141c76e-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:29:19.659303 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.659300    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqr47\" (UniqueName: \"kubernetes.io/projected/830555b4-08e7-4df2-bc3f-2bbc3141c76e-kube-api-access-zqr47\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:29:19.659498 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.659309    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/830555b4-08e7-4df2-bc3f-2bbc3141c76e-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:29:19.760061 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.759994    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58592270-db06-499b-ab98-119e6331b69a-plugins-config-volume\") pod \"58592270-db06-499b-ab98-119e6331b69a\" (UID: \"58592270-db06-499b-ab98-119e6331b69a\") "
Jul 28 12:29:19.760061 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.760041    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgmzv\" (UniqueName: \"kubernetes.io/projected/58592270-db06-499b-ab98-119e6331b69a-kube-api-access-bgmzv\") pod \"58592270-db06-499b-ab98-119e6331b69a\" (UID: \"58592270-db06-499b-ab98-119e6331b69a\") "
Jul 28 12:29:19.760216 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.760092    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7f5jf\" (UniqueName: \"kubernetes.io/projected/35cb4004-f092-4b19-ae7c-5b582bb7f555-kube-api-access-7f5jf\") pod \"35cb4004-f092-4b19-ae7c-5b582bb7f555\" (UID: \"35cb4004-f092-4b19-ae7c-5b582bb7f555\") "
Jul 28 12:29:19.760216 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.760117    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35cb4004-f092-4b19-ae7c-5b582bb7f555-plugins-config-volume\") pod \"35cb4004-f092-4b19-ae7c-5b582bb7f555\" (UID: \"35cb4004-f092-4b19-ae7c-5b582bb7f555\") "
Jul 28 12:29:19.760535 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.760483    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58592270-db06-499b-ab98-119e6331b69a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "58592270-db06-499b-ab98-119e6331b69a" (UID: "58592270-db06-499b-ab98-119e6331b69a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:29:19.760669 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.760533    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35cb4004-f092-4b19-ae7c-5b582bb7f555-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "35cb4004-f092-4b19-ae7c-5b582bb7f555" (UID: "35cb4004-f092-4b19-ae7c-5b582bb7f555"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:29:19.762208 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.762180    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58592270-db06-499b-ab98-119e6331b69a-kube-api-access-bgmzv" (OuterVolumeSpecName: "kube-api-access-bgmzv") pod "58592270-db06-499b-ab98-119e6331b69a" (UID: "58592270-db06-499b-ab98-119e6331b69a"). InnerVolumeSpecName "kube-api-access-bgmzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:29:19.762293 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.762219    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cb4004-f092-4b19-ae7c-5b582bb7f555-kube-api-access-7f5jf" (OuterVolumeSpecName: "kube-api-access-7f5jf") pod "35cb4004-f092-4b19-ae7c-5b582bb7f555" (UID: "35cb4004-f092-4b19-ae7c-5b582bb7f555"). InnerVolumeSpecName "kube-api-access-7f5jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:29:19.860891 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.860866    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f5jf\" (UniqueName: \"kubernetes.io/projected/35cb4004-f092-4b19-ae7c-5b582bb7f555-kube-api-access-7f5jf\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:29:19.860891 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.860889    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35cb4004-f092-4b19-ae7c-5b582bb7f555-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:29:19.861021 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.860906    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58592270-db06-499b-ab98-119e6331b69a-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:29:19.861021 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.860921    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgmzv\" (UniqueName: \"kubernetes.io/projected/58592270-db06-499b-ab98-119e6331b69a-kube-api-access-bgmzv\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:29:19.975316 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.975289    2561 generic.go:356] "Generic (PLEG): container finished" podID="830555b4-08e7-4df2-bc3f-2bbc3141c76e" containerID="f3cc8f81402bc7aa3e46b2ab27c91dcdc19b201a9848758a936cd68beaabc921" exitCode=0
Jul 28 12:29:19.975431 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.975342    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"
Jul 28 12:29:19.975431 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.975370    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx" event={"ID":"830555b4-08e7-4df2-bc3f-2bbc3141c76e","Type":"ContainerDied","Data":"f3cc8f81402bc7aa3e46b2ab27c91dcdc19b201a9848758a936cd68beaabc921"}
Jul 28 12:29:19.975556 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.975435    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx" event={"ID":"830555b4-08e7-4df2-bc3f-2bbc3141c76e","Type":"ContainerDied","Data":"b59798d3cc480f813b57b11d43b61b84f2792d19f0504e30423402e48d16b02a"}
Jul 28 12:29:19.975556 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.975467    2561 scope.go:117] "RemoveContainer" containerID="f3cc8f81402bc7aa3e46b2ab27c91dcdc19b201a9848758a936cd68beaabc921"
Jul 28 12:29:19.976689 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.976658    2561 generic.go:356] "Generic (PLEG): container finished" podID="58592270-db06-499b-ab98-119e6331b69a" containerID="60999e73ba0b3a29abacb3472dc38cbc91f05b79c370d77e62670aa245c4fdb4" exitCode=0
Jul 28 12:29:19.976767 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.976706    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6"
Jul 28 12:29:19.976767 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.976747    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6" event={"ID":"58592270-db06-499b-ab98-119e6331b69a","Type":"ContainerDied","Data":"60999e73ba0b3a29abacb3472dc38cbc91f05b79c370d77e62670aa245c4fdb4"}
Jul 28 12:29:19.976870 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.976799    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6" event={"ID":"58592270-db06-499b-ab98-119e6331b69a","Type":"ContainerDied","Data":"b6175e05cde40215cba0894c15723adbea5d2c7351249f1b4659626bbd5a0706"}
Jul 28 12:29:19.978844 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.978821    2561 generic.go:356] "Generic (PLEG): container finished" podID="35cb4004-f092-4b19-ae7c-5b582bb7f555" containerID="b8083a2b6f4669b9c0cb8a58d26c92b8db2e428201acf1676491e526b1ffc461" exitCode=0
Jul 28 12:29:19.978962 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.978859    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54"
Jul 28 12:29:19.978962 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.978905    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54" event={"ID":"35cb4004-f092-4b19-ae7c-5b582bb7f555","Type":"ContainerDied","Data":"b8083a2b6f4669b9c0cb8a58d26c92b8db2e428201acf1676491e526b1ffc461"}
Jul 28 12:29:19.978962 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.978940    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54" event={"ID":"35cb4004-f092-4b19-ae7c-5b582bb7f555","Type":"ContainerDied","Data":"2a6603b347c919fce041448907fa4af93884c16f0ebe672d9cec6bc57f7772cd"}
Jul 28 12:29:19.986637 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.986230    2561 scope.go:117] "RemoveContainer" containerID="f3cc8f81402bc7aa3e46b2ab27c91dcdc19b201a9848758a936cd68beaabc921"
Jul 28 12:29:19.986941 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:29:19.986914    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3cc8f81402bc7aa3e46b2ab27c91dcdc19b201a9848758a936cd68beaabc921\": container with ID starting with f3cc8f81402bc7aa3e46b2ab27c91dcdc19b201a9848758a936cd68beaabc921 not found: ID does not exist" containerID="f3cc8f81402bc7aa3e46b2ab27c91dcdc19b201a9848758a936cd68beaabc921"
Jul 28 12:29:19.987040 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.986950    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3cc8f81402bc7aa3e46b2ab27c91dcdc19b201a9848758a936cd68beaabc921"} err="failed to get container status \"f3cc8f81402bc7aa3e46b2ab27c91dcdc19b201a9848758a936cd68beaabc921\": rpc error: code = NotFound desc = could not find container \"f3cc8f81402bc7aa3e46b2ab27c91dcdc19b201a9848758a936cd68beaabc921\": container with ID starting with f3cc8f81402bc7aa3e46b2ab27c91dcdc19b201a9848758a936cd68beaabc921 not found: ID does not exist"
Jul 28 12:29:19.987040 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.986975    2561 scope.go:117] "RemoveContainer" containerID="60999e73ba0b3a29abacb3472dc38cbc91f05b79c370d77e62670aa245c4fdb4"
Jul 28 12:29:19.999979 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:19.999962    2561 scope.go:117] "RemoveContainer" containerID="60999e73ba0b3a29abacb3472dc38cbc91f05b79c370d77e62670aa245c4fdb4"
Jul 28 12:29:20.000229 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:29:20.000211    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60999e73ba0b3a29abacb3472dc38cbc91f05b79c370d77e62670aa245c4fdb4\": container with ID starting with 60999e73ba0b3a29abacb3472dc38cbc91f05b79c370d77e62670aa245c4fdb4 not found: ID does not exist" containerID="60999e73ba0b3a29abacb3472dc38cbc91f05b79c370d77e62670aa245c4fdb4"
Jul 28 12:29:20.000300 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:20.000241    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60999e73ba0b3a29abacb3472dc38cbc91f05b79c370d77e62670aa245c4fdb4"} err="failed to get container status \"60999e73ba0b3a29abacb3472dc38cbc91f05b79c370d77e62670aa245c4fdb4\": rpc error: code = NotFound desc = could not find container \"60999e73ba0b3a29abacb3472dc38cbc91f05b79c370d77e62670aa245c4fdb4\": container with ID starting with 60999e73ba0b3a29abacb3472dc38cbc91f05b79c370d77e62670aa245c4fdb4 not found: ID does not exist"
Jul 28 12:29:20.000300 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:20.000264    2561 scope.go:117] "RemoveContainer" containerID="b8083a2b6f4669b9c0cb8a58d26c92b8db2e428201acf1676491e526b1ffc461"
Jul 28 12:29:20.007644 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:20.007624    2561 scope.go:117] "RemoveContainer" containerID="b8083a2b6f4669b9c0cb8a58d26c92b8db2e428201acf1676491e526b1ffc461"
Jul 28 12:29:20.007942 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:29:20.007922    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8083a2b6f4669b9c0cb8a58d26c92b8db2e428201acf1676491e526b1ffc461\": container with ID starting with b8083a2b6f4669b9c0cb8a58d26c92b8db2e428201acf1676491e526b1ffc461 not found: ID does not exist" containerID="b8083a2b6f4669b9c0cb8a58d26c92b8db2e428201acf1676491e526b1ffc461"
Jul 28 12:29:20.008008 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:20.007953    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8083a2b6f4669b9c0cb8a58d26c92b8db2e428201acf1676491e526b1ffc461"} err="failed to get container status \"b8083a2b6f4669b9c0cb8a58d26c92b8db2e428201acf1676491e526b1ffc461\": rpc error: code = NotFound desc = could not find container \"b8083a2b6f4669b9c0cb8a58d26c92b8db2e428201acf1676491e526b1ffc461\": container with ID starting with b8083a2b6f4669b9c0cb8a58d26c92b8db2e428201acf1676491e526b1ffc461 not found: ID does not exist"
Jul 28 12:29:20.010996 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:20.010947    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"]
Jul 28 12:29:20.017389 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:20.017370    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4d5cfe-7b57f5784d-k2fqx"]
Jul 28 12:29:20.027888 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:20.027866    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54"]
Jul 28 12:29:20.034567 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:20.034547    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4d5cfe-748fcf654b-khs54"]
Jul 28 12:29:20.048799 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:20.048781    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6"]
Jul 28 12:29:20.055329 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:20.055310    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4d5cfe-655f67b776-9vmx6"]
Jul 28 12:29:20.545647 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:20.545614    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35cb4004-f092-4b19-ae7c-5b582bb7f555" path="/var/lib/kubelet/pods/35cb4004-f092-4b19-ae7c-5b582bb7f555/volumes"
Jul 28 12:29:20.546018 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:20.545964    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58592270-db06-499b-ab98-119e6331b69a" path="/var/lib/kubelet/pods/58592270-db06-499b-ab98-119e6331b69a/volumes"
Jul 28 12:29:20.546294 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:20.546281    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="830555b4-08e7-4df2-bc3f-2bbc3141c76e" path="/var/lib/kubelet/pods/830555b4-08e7-4df2-bc3f-2bbc3141c76e/volumes"
Jul 28 12:29:22.752205 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:22.752171    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:29:31.243160 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243115    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e7af0c-lscbm"]
Jul 28 12:29:31.243667 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243593    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b39f2b10-84e5-433d-a9ee-c4e68f9f7650" containerName="maas-api"
Jul 28 12:29:31.243667 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243613    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b39f2b10-84e5-433d-a9ee-c4e68f9f7650" containerName="maas-api"
Jul 28 12:29:31.243667 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243630    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08294600-509c-4997-9eea-c4e1abeedb3e" containerName="payload-pre-processing"
Jul 28 12:29:31.243667 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243639    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="08294600-509c-4997-9eea-c4e1abeedb3e" containerName="payload-pre-processing"
Jul 28 12:29:31.243667 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243661    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35cb4004-f092-4b19-ae7c-5b582bb7f555" containerName="payload-processing"
Jul 28 12:29:31.243667 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243670    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="35cb4004-f092-4b19-ae7c-5b582bb7f555" containerName="payload-processing"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243679    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="830555b4-08e7-4df2-bc3f-2bbc3141c76e" containerName="maas-api"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243687    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="830555b4-08e7-4df2-bc3f-2bbc3141c76e" containerName="maas-api"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243699    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58592270-db06-499b-ab98-119e6331b69a" containerName="payload-pre-processing"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243707    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="58592270-db06-499b-ab98-119e6331b69a" containerName="payload-pre-processing"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243717    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8474023-b495-4710-9062-6e008ff5c712" containerName="payload-processing"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243726    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8474023-b495-4710-9062-6e008ff5c712" containerName="payload-processing"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243736    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6624dc-ed2a-460c-a91d-dec3548ee355" containerName="main"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243745    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6624dc-ed2a-460c-a91d-dec3548ee355" containerName="main"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243756    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e2a1a6-9ac3-4969-a0cf-57482a3964fe" containerName="payload-pre-processing"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243764    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e2a1a6-9ac3-4969-a0cf-57482a3964fe" containerName="payload-pre-processing"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243784    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbd0f8c3-2b81-4319-b624-5104697eb0ee" containerName="payload-pre-processing"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243792    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbd0f8c3-2b81-4319-b624-5104697eb0ee" containerName="payload-pre-processing"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243873    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="58592270-db06-499b-ab98-119e6331b69a" containerName="payload-pre-processing"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243884    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbd0f8c3-2b81-4319-b624-5104697eb0ee" containerName="payload-pre-processing"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243892    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e2a1a6-9ac3-4969-a0cf-57482a3964fe" containerName="payload-pre-processing"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243902    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="35cb4004-f092-4b19-ae7c-5b582bb7f555" containerName="payload-processing"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243911    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b39f2b10-84e5-433d-a9ee-c4e68f9f7650" containerName="maas-api"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243920    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a6624dc-ed2a-460c-a91d-dec3548ee355" containerName="main"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243930    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="830555b4-08e7-4df2-bc3f-2bbc3141c76e" containerName="maas-api"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243938    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8474023-b495-4710-9062-6e008ff5c712" containerName="payload-processing"
Jul 28 12:29:31.243993 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.243947    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="08294600-509c-4997-9eea-c4e1abeedb3e" containerName="payload-pre-processing"
Jul 28 12:29:31.248922 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.248900    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e7af0c-lscbm"
Jul 28 12:29:31.251131 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.251104    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ts4x9\""
Jul 28 12:29:31.252436 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.252411    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e7af0c-lscbm"]
Jul 28 12:29:31.362193 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.362155    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db04c82c-ba2c-4d23-a3b4-d86dcf64cf08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-e7af0c-lscbm\" (UID: \"db04c82c-ba2c-4d23-a3b4-d86dcf64cf08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e7af0c-lscbm"
Jul 28 12:29:31.463294 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.463262    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db04c82c-ba2c-4d23-a3b4-d86dcf64cf08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-e7af0c-lscbm\" (UID: \"db04c82c-ba2c-4d23-a3b4-d86dcf64cf08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e7af0c-lscbm"
Jul 28 12:29:31.463929 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.463905    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db04c82c-ba2c-4d23-a3b4-d86dcf64cf08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-e7af0c-lscbm\" (UID: \"db04c82c-ba2c-4d23-a3b4-d86dcf64cf08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e7af0c-lscbm"
Jul 28 12:29:31.560245 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.560209    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e7af0c-lscbm"
Jul 28 12:29:31.689868 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:31.689758    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e7af0c-lscbm"]
Jul 28 12:29:31.692084 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:29:31.692042    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb04c82c_ba2c_4d23_a3b4_d86dcf64cf08.slice/crio-706a7aba71b6420fe236730f699f602f321302e9f81f732ac3f21bb62ff31dee WatchSource:0}: Error finding container 706a7aba71b6420fe236730f699f602f321302e9f81f732ac3f21bb62ff31dee: Status 404 returned error can't find the container with id 706a7aba71b6420fe236730f699f602f321302e9f81f732ac3f21bb62ff31dee
Jul 28 12:29:32.034573 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:32.034543    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e7af0c-lscbm" event={"ID":"db04c82c-ba2c-4d23-a3b4-d86dcf64cf08","Type":"ContainerStarted","Data":"e7226496b81a97e754cf9a7c3b21e3d925dff889308af9a81efdb328153fd57c"}
Jul 28 12:29:32.034729 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:32.034612    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e7af0c-lscbm" event={"ID":"db04c82c-ba2c-4d23-a3b4-d86dcf64cf08","Type":"ContainerStarted","Data":"706a7aba71b6420fe236730f699f602f321302e9f81f732ac3f21bb62ff31dee"}
Jul 28 12:29:32.052222 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:32.052176    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e7af0c-lscbm" podStartSLOduration=1.05216349 podStartE2EDuration="1.05216349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:29:32.051446325 +0000 UTC m=+3523.910102324" watchObservedRunningTime="2026-07-28 12:29:32.05216349 +0000 UTC m=+3523.910819488"
Jul 28 12:29:33.040802 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:33.040761    2561 generic.go:356] "Generic (PLEG): container finished" podID="db04c82c-ba2c-4d23-a3b4-d86dcf64cf08" containerID="e7226496b81a97e754cf9a7c3b21e3d925dff889308af9a81efdb328153fd57c" exitCode=0
Jul 28 12:29:33.041169 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:33.040839    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e7af0c-lscbm" event={"ID":"db04c82c-ba2c-4d23-a3b4-d86dcf64cf08","Type":"ContainerDied","Data":"e7226496b81a97e754cf9a7c3b21e3d925dff889308af9a81efdb328153fd57c"}
Jul 28 12:29:34.177815 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:34.177789    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e7af0c-lscbm"
Jul 28 12:29:34.288243 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:34.288213    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db04c82c-ba2c-4d23-a3b4-d86dcf64cf08-maas-api-service-ca\") pod \"db04c82c-ba2c-4d23-a3b4-d86dcf64cf08\" (UID: \"db04c82c-ba2c-4d23-a3b4-d86dcf64cf08\") "
Jul 28 12:29:34.288550 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:34.288527    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db04c82c-ba2c-4d23-a3b4-d86dcf64cf08-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "db04c82c-ba2c-4d23-a3b4-d86dcf64cf08" (UID: "db04c82c-ba2c-4d23-a3b4-d86dcf64cf08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:29:34.389082 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:34.389054    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db04c82c-ba2c-4d23-a3b4-d86dcf64cf08-maas-api-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:29:35.052860 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:35.052832    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e7af0c-lscbm"
Jul 28 12:29:35.053027 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:35.052861    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e7af0c-lscbm" event={"ID":"db04c82c-ba2c-4d23-a3b4-d86dcf64cf08","Type":"ContainerDied","Data":"706a7aba71b6420fe236730f699f602f321302e9f81f732ac3f21bb62ff31dee"}
Jul 28 12:29:35.053027 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:35.052908    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="706a7aba71b6420fe236730f699f602f321302e9f81f732ac3f21bb62ff31dee"
Jul 28 12:29:41.737549 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:41.737516    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"]
Jul 28 12:29:41.738045 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:41.737813    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq" podUID="667a5e55-9c49-4cb7-80e2-7306c9382006" containerName="maas-api" containerID="cri-o://8d598b641fc0289486bc901b1a7596a5c032fb2e11a8decd0d699d68c2e87c35" gracePeriod=30
Jul 28 12:29:41.792360 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:41.792324    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk"]
Jul 28 12:29:41.792597 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:41.792536    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk" podUID="fc8fba81-cbee-4737-9f72-0fed7faa029e" containerName="payload-processing" containerID="cri-o://3a33ae8acf304a0106f36acb24c92e54ba96ea0148438091b3bec94ca69adcb2" gracePeriod=30
Jul 28 12:29:41.799833 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:41.799806    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz"]
Jul 28 12:29:41.800016 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:41.799989    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz" podUID="25bef550-03be-4795-894a-2dca4589dac8" containerName="payload-pre-processing" containerID="cri-o://6106c851c6e2a46c4b3f725b3c59464b48d4c47099b4675c366bb09d93e4e921" gracePeriod=30
Jul 28 12:29:42.043513 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.043486    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"
Jul 28 12:29:42.046400 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.046379    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/667a5e55-9c49-4cb7-80e2-7306c9382006-maas-api-tls\") pod \"667a5e55-9c49-4cb7-80e2-7306c9382006\" (UID: \"667a5e55-9c49-4cb7-80e2-7306c9382006\") "
Jul 28 12:29:42.046523 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.046439    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/667a5e55-9c49-4cb7-80e2-7306c9382006-openshift-service-ca\") pod \"667a5e55-9c49-4cb7-80e2-7306c9382006\" (UID: \"667a5e55-9c49-4cb7-80e2-7306c9382006\") "
Jul 28 12:29:42.046523 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.046509    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxdgs\" (UniqueName: \"kubernetes.io/projected/667a5e55-9c49-4cb7-80e2-7306c9382006-kube-api-access-qxdgs\") pod \"667a5e55-9c49-4cb7-80e2-7306c9382006\" (UID: \"667a5e55-9c49-4cb7-80e2-7306c9382006\") "
Jul 28 12:29:42.046901 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.046862    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/667a5e55-9c49-4cb7-80e2-7306c9382006-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "667a5e55-9c49-4cb7-80e2-7306c9382006" (UID: "667a5e55-9c49-4cb7-80e2-7306c9382006"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:29:42.050715 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.050691    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667a5e55-9c49-4cb7-80e2-7306c9382006-kube-api-access-qxdgs" (OuterVolumeSpecName: "kube-api-access-qxdgs") pod "667a5e55-9c49-4cb7-80e2-7306c9382006" (UID: "667a5e55-9c49-4cb7-80e2-7306c9382006"). InnerVolumeSpecName "kube-api-access-qxdgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:29:42.052365 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.052344    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/667a5e55-9c49-4cb7-80e2-7306c9382006-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "667a5e55-9c49-4cb7-80e2-7306c9382006" (UID: "667a5e55-9c49-4cb7-80e2-7306c9382006"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:29:42.080629 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.080598    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk"
Jul 28 12:29:42.083848 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.083822    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz"
Jul 28 12:29:42.086913 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.086886    2561 generic.go:356] "Generic (PLEG): container finished" podID="25bef550-03be-4795-894a-2dca4589dac8" containerID="6106c851c6e2a46c4b3f725b3c59464b48d4c47099b4675c366bb09d93e4e921" exitCode=0
Jul 28 12:29:42.087035 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.086929    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz"
Jul 28 12:29:42.087035 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.086964    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz" event={"ID":"25bef550-03be-4795-894a-2dca4589dac8","Type":"ContainerDied","Data":"6106c851c6e2a46c4b3f725b3c59464b48d4c47099b4675c366bb09d93e4e921"}
Jul 28 12:29:42.087035 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.087025    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz" event={"ID":"25bef550-03be-4795-894a-2dca4589dac8","Type":"ContainerDied","Data":"6477b952f578a5fa5bb9456a56f476a416b49c20da960815740b14828809ca74"}
Jul 28 12:29:42.087192 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.087045    2561 scope.go:117] "RemoveContainer" containerID="6106c851c6e2a46c4b3f725b3c59464b48d4c47099b4675c366bb09d93e4e921"
Jul 28 12:29:42.088514 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.088489    2561 generic.go:356] "Generic (PLEG): container finished" podID="fc8fba81-cbee-4737-9f72-0fed7faa029e" containerID="3a33ae8acf304a0106f36acb24c92e54ba96ea0148438091b3bec94ca69adcb2" exitCode=0
Jul 28 12:29:42.088647 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.088555    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk"
Jul 28 12:29:42.088647 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.088606    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk" event={"ID":"fc8fba81-cbee-4737-9f72-0fed7faa029e","Type":"ContainerDied","Data":"3a33ae8acf304a0106f36acb24c92e54ba96ea0148438091b3bec94ca69adcb2"}
Jul 28 12:29:42.088759 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.088662    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk" event={"ID":"fc8fba81-cbee-4737-9f72-0fed7faa029e","Type":"ContainerDied","Data":"43d06f4699e207d76d5931f54d9038da842e2beb8a578b83c693ef8ff6765720"}
Jul 28 12:29:42.090004 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.089984    2561 generic.go:356] "Generic (PLEG): container finished" podID="667a5e55-9c49-4cb7-80e2-7306c9382006" containerID="8d598b641fc0289486bc901b1a7596a5c032fb2e11a8decd0d699d68c2e87c35" exitCode=0
Jul 28 12:29:42.090099 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.090015    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq" event={"ID":"667a5e55-9c49-4cb7-80e2-7306c9382006","Type":"ContainerDied","Data":"8d598b641fc0289486bc901b1a7596a5c032fb2e11a8decd0d699d68c2e87c35"}
Jul 28 12:29:42.090099 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.090039    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"
Jul 28 12:29:42.090099 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.090069    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq" event={"ID":"667a5e55-9c49-4cb7-80e2-7306c9382006","Type":"ContainerDied","Data":"78dc76b74dca63a80c3a33f98e5a236c10b44bba5b5eddf1e88d4cdd2b1b0132"}
Jul 28 12:29:42.098365 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.098321    2561 scope.go:117] "RemoveContainer" containerID="6106c851c6e2a46c4b3f725b3c59464b48d4c47099b4675c366bb09d93e4e921"
Jul 28 12:29:42.098936 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:29:42.098780    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6106c851c6e2a46c4b3f725b3c59464b48d4c47099b4675c366bb09d93e4e921\": container with ID starting with 6106c851c6e2a46c4b3f725b3c59464b48d4c47099b4675c366bb09d93e4e921 not found: ID does not exist" containerID="6106c851c6e2a46c4b3f725b3c59464b48d4c47099b4675c366bb09d93e4e921"
Jul 28 12:29:42.098936 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.098822    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6106c851c6e2a46c4b3f725b3c59464b48d4c47099b4675c366bb09d93e4e921"} err="failed to get container status \"6106c851c6e2a46c4b3f725b3c59464b48d4c47099b4675c366bb09d93e4e921\": rpc error: code = NotFound desc = could not find container \"6106c851c6e2a46c4b3f725b3c59464b48d4c47099b4675c366bb09d93e4e921\": container with ID starting with 6106c851c6e2a46c4b3f725b3c59464b48d4c47099b4675c366bb09d93e4e921 not found: ID does not exist"
Jul 28 12:29:42.098936 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.098850    2561 scope.go:117] "RemoveContainer" containerID="3a33ae8acf304a0106f36acb24c92e54ba96ea0148438091b3bec94ca69adcb2"
Jul 28 12:29:42.109438 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.109414    2561 scope.go:117] "RemoveContainer" containerID="3a33ae8acf304a0106f36acb24c92e54ba96ea0148438091b3bec94ca69adcb2"
Jul 28 12:29:42.109746 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:29:42.109726    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a33ae8acf304a0106f36acb24c92e54ba96ea0148438091b3bec94ca69adcb2\": container with ID starting with 3a33ae8acf304a0106f36acb24c92e54ba96ea0148438091b3bec94ca69adcb2 not found: ID does not exist" containerID="3a33ae8acf304a0106f36acb24c92e54ba96ea0148438091b3bec94ca69adcb2"
Jul 28 12:29:42.109825 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.109756    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a33ae8acf304a0106f36acb24c92e54ba96ea0148438091b3bec94ca69adcb2"} err="failed to get container status \"3a33ae8acf304a0106f36acb24c92e54ba96ea0148438091b3bec94ca69adcb2\": rpc error: code = NotFound desc = could not find container \"3a33ae8acf304a0106f36acb24c92e54ba96ea0148438091b3bec94ca69adcb2\": container with ID starting with 3a33ae8acf304a0106f36acb24c92e54ba96ea0148438091b3bec94ca69adcb2 not found: ID does not exist"
Jul 28 12:29:42.109825 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.109776    2561 scope.go:117] "RemoveContainer" containerID="8d598b641fc0289486bc901b1a7596a5c032fb2e11a8decd0d699d68c2e87c35"
Jul 28 12:29:42.119357 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.119336    2561 scope.go:117] "RemoveContainer" containerID="8d598b641fc0289486bc901b1a7596a5c032fb2e11a8decd0d699d68c2e87c35"
Jul 28 12:29:42.119668 ip-10-0-141-155 kubenswrapper[2561]: E0728 12:29:42.119641    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d598b641fc0289486bc901b1a7596a5c032fb2e11a8decd0d699d68c2e87c35\": container with ID starting with 8d598b641fc0289486bc901b1a7596a5c032fb2e11a8decd0d699d68c2e87c35 not found: ID does not exist" containerID="8d598b641fc0289486bc901b1a7596a5c032fb2e11a8decd0d699d68c2e87c35"
Jul 28 12:29:42.119762 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.119683    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d598b641fc0289486bc901b1a7596a5c032fb2e11a8decd0d699d68c2e87c35"} err="failed to get container status \"8d598b641fc0289486bc901b1a7596a5c032fb2e11a8decd0d699d68c2e87c35\": rpc error: code = NotFound desc = could not find container \"8d598b641fc0289486bc901b1a7596a5c032fb2e11a8decd0d699d68c2e87c35\": container with ID starting with 8d598b641fc0289486bc901b1a7596a5c032fb2e11a8decd0d699d68c2e87c35 not found: ID does not exist"
Jul 28 12:29:42.134126 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.134096    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"]
Jul 28 12:29:42.140662 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.140638    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e7af0c-5fb9d9f95b-mh9tq"]
Jul 28 12:29:42.147055 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.147036    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kw9kb\" (UniqueName: \"kubernetes.io/projected/fc8fba81-cbee-4737-9f72-0fed7faa029e-kube-api-access-kw9kb\") pod \"fc8fba81-cbee-4737-9f72-0fed7faa029e\" (UID: \"fc8fba81-cbee-4737-9f72-0fed7faa029e\") "
Jul 28 12:29:42.147142 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.147072    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhw4g\" (UniqueName: \"kubernetes.io/projected/25bef550-03be-4795-894a-2dca4589dac8-kube-api-access-fhw4g\") pod \"25bef550-03be-4795-894a-2dca4589dac8\" (UID: \"25bef550-03be-4795-894a-2dca4589dac8\") "
Jul 28 12:29:42.147142 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.147109    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25bef550-03be-4795-894a-2dca4589dac8-plugins-config-volume\") pod \"25bef550-03be-4795-894a-2dca4589dac8\" (UID: \"25bef550-03be-4795-894a-2dca4589dac8\") "
Jul 28 12:29:42.147241 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.147195    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8fba81-cbee-4737-9f72-0fed7faa029e-plugins-config-volume\") pod \"fc8fba81-cbee-4737-9f72-0fed7faa029e\" (UID: \"fc8fba81-cbee-4737-9f72-0fed7faa029e\") "
Jul 28 12:29:42.147412 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.147393    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/667a5e55-9c49-4cb7-80e2-7306c9382006-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:29:42.147478 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.147418    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxdgs\" (UniqueName: \"kubernetes.io/projected/667a5e55-9c49-4cb7-80e2-7306c9382006-kube-api-access-qxdgs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:29:42.147478 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.147434    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/667a5e55-9c49-4cb7-80e2-7306c9382006-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:29:42.147592 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.147512    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8fba81-cbee-4737-9f72-0fed7faa029e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc8fba81-cbee-4737-9f72-0fed7faa029e" (UID: "fc8fba81-cbee-4737-9f72-0fed7faa029e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:29:42.147592 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.147513    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25bef550-03be-4795-894a-2dca4589dac8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "25bef550-03be-4795-894a-2dca4589dac8" (UID: "25bef550-03be-4795-894a-2dca4589dac8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:29:42.149236 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.149218    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25bef550-03be-4795-894a-2dca4589dac8-kube-api-access-fhw4g" (OuterVolumeSpecName: "kube-api-access-fhw4g") pod "25bef550-03be-4795-894a-2dca4589dac8" (UID: "25bef550-03be-4795-894a-2dca4589dac8"). InnerVolumeSpecName "kube-api-access-fhw4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:29:42.149298 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.149260    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8fba81-cbee-4737-9f72-0fed7faa029e-kube-api-access-kw9kb" (OuterVolumeSpecName: "kube-api-access-kw9kb") pod "fc8fba81-cbee-4737-9f72-0fed7faa029e" (UID: "fc8fba81-cbee-4737-9f72-0fed7faa029e"). InnerVolumeSpecName "kube-api-access-kw9kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:29:42.247902 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.247863    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8fba81-cbee-4737-9f72-0fed7faa029e-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:29:42.247902 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.247893    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw9kb\" (UniqueName: \"kubernetes.io/projected/fc8fba81-cbee-4737-9f72-0fed7faa029e-kube-api-access-kw9kb\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:29:42.247902 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.247903    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhw4g\" (UniqueName: \"kubernetes.io/projected/25bef550-03be-4795-894a-2dca4589dac8-kube-api-access-fhw4g\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:29:42.247902 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.247911    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25bef550-03be-4795-894a-2dca4589dac8-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\""
Jul 28 12:29:42.411740 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.411713    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz"]
Jul 28 12:29:42.416148 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.416120    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-e7af0c-5f976bd7c6-652zz"]
Jul 28 12:29:42.427034 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.427009    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk"]
Jul 28 12:29:42.428535 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.428516    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-e7af0c-74fcc757d-9ftdk"]
Jul 28 12:29:42.546381 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.546349    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25bef550-03be-4795-894a-2dca4589dac8" path="/var/lib/kubelet/pods/25bef550-03be-4795-894a-2dca4589dac8/volumes"
Jul 28 12:29:42.546760 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.546745    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="667a5e55-9c49-4cb7-80e2-7306c9382006" path="/var/lib/kubelet/pods/667a5e55-9c49-4cb7-80e2-7306c9382006/volumes"
Jul 28 12:29:42.547069 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:42.547057    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc8fba81-cbee-4737-9f72-0fed7faa029e" path="/var/lib/kubelet/pods/fc8fba81-cbee-4737-9f72-0fed7faa029e/volumes"
Jul 28 12:29:44.534191 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:44.534154    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6t8t4"]
Jul 28 12:29:46.339750 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:46.339716    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e7af0c-lscbm"]
Jul 28 12:29:46.343097 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:46.343072    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e7af0c-lscbm"]
Jul 28 12:29:46.546193 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:46.546158    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db04c82c-ba2c-4d23-a3b4-d86dcf64cf08" path="/var/lib/kubelet/pods/db04c82c-ba2c-4d23-a3b4-d86dcf64cf08/volumes"
Jul 28 12:29:59.972956 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:29:59.972921    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-l6wtq_d7032b5a-b6d6-432a-9608-41553f3bf317/manager/0.log"
Jul 28 12:30:00.134211 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:00.134167    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754030-677jm"]
Jul 28 12:30:00.134610 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:00.134568    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc8fba81-cbee-4737-9f72-0fed7faa029e" containerName="payload-processing"
Jul 28 12:30:00.134610 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:00.134603    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc8fba81-cbee-4737-9f72-0fed7faa029e" containerName="payload-processing"
Jul 28 12:30:00.134767 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:00.134623    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25bef550-03be-4795-894a-2dca4589dac8" containerName="payload-pre-processing"
Jul 28 12:30:00.134767 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:00.134631    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="25bef550-03be-4795-894a-2dca4589dac8" containerName="payload-pre-processing"
Jul 28 12:30:00.134767 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:00.134648    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db04c82c-ba2c-4d23-a3b4-d86dcf64cf08" containerName="revoke-keys"
Jul 28 12:30:00.134767 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:00.134658    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="db04c82c-ba2c-4d23-a3b4-d86dcf64cf08" containerName="revoke-keys"
Jul 28 12:30:00.134767 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:00.134670    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667a5e55-9c49-4cb7-80e2-7306c9382006" containerName="maas-api"
Jul 28 12:30:00.134767 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:00.134678    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="667a5e55-9c49-4cb7-80e2-7306c9382006" containerName="maas-api"
Jul 28 12:30:00.134767 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:00.134758    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="25bef550-03be-4795-894a-2dca4589dac8" containerName="payload-pre-processing"
Jul 28 12:30:00.134767 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:00.134770    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="667a5e55-9c49-4cb7-80e2-7306c9382006" containerName="maas-api"
Jul 28 12:30:00.135160 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:00.134780    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc8fba81-cbee-4737-9f72-0fed7faa029e" containerName="payload-processing"
Jul 28 12:30:00.135160 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:00.134791    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="db04c82c-ba2c-4d23-a3b4-d86dcf64cf08" containerName="revoke-keys"
Jul 28 12:30:00.139851 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:00.139828    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754030-677jm"
Jul 28 12:30:00.142120 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:00.142096    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ts4x9\""
Jul 28 12:30:00.144001 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:00.143974    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754030-677jm"]
Jul 28 12:30:00.150853 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:00.150835    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754030-677jm"
Jul 28 12:30:00.280286 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:00.280241    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754030-677jm"]
Jul 28 12:30:00.285557 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:30:00.285528    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f630f2_c9da_485c_b474_6a5eb53d70bd.slice/crio-24efd222a72ac717b3505fcc393f3636e733b1c9dc79e6e91b9fd1f8fa87278d WatchSource:0}: Error finding container 24efd222a72ac717b3505fcc393f3636e733b1c9dc79e6e91b9fd1f8fa87278d: Status 404 returned error can't find the container with id 24efd222a72ac717b3505fcc393f3636e733b1c9dc79e6e91b9fd1f8fa87278d
Jul 28 12:30:00.603574 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:00.603548    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6755d7577-8fmcq_463fa1da-5fff-4de1-815b-2f76070d5165/copy-manifests/0.log"
Jul 28 12:30:00.611177 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:00.611157    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6755d7577-8fmcq_463fa1da-5fff-4de1-815b-2f76070d5165/manager/0.log"
Jul 28 12:30:01.178038 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:01.178008    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754030-677jm" event={"ID":"f6f630f2-c9da-485c-b474-6a5eb53d70bd","Type":"ContainerStarted","Data":"2d4f1e8bf348f55668d54e60a635c26f046709bf34c39a012c0a692025b7f380"}
Jul 28 12:30:01.178368 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:01.178057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754030-677jm" event={"ID":"f6f630f2-c9da-485c-b474-6a5eb53d70bd","Type":"ContainerStarted","Data":"24efd222a72ac717b3505fcc393f3636e733b1c9dc79e6e91b9fd1f8fa87278d"}
Jul 28 12:30:01.405640 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:01.405597    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754000-f6gf2_d3c02af9-91b9-48e9-8b4d-109366e7203a/cleanup/0.log"
Jul 28 12:30:01.661349 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:01.661311    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754030-677jm_f6f630f2-c9da-485c-b474-6a5eb53d70bd/cleanup/0.log"
Jul 28 12:30:02.184193 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:02.184160    2561 generic.go:356] "Generic (PLEG): container finished" podID="f6f630f2-c9da-485c-b474-6a5eb53d70bd" containerID="2d4f1e8bf348f55668d54e60a635c26f046709bf34c39a012c0a692025b7f380" exitCode=0
Jul 28 12:30:02.184602 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:02.184205    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754030-677jm" event={"ID":"f6f630f2-c9da-485c-b474-6a5eb53d70bd","Type":"ContainerDied","Data":"2d4f1e8bf348f55668d54e60a635c26f046709bf34c39a012c0a692025b7f380"}
Jul 28 12:30:02.311290 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:02.311270    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754030-677jm"
Jul 28 12:30:03.189691 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:03.189664    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754030-677jm"
Jul 28 12:30:03.189691 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:03.189678    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754030-677jm" event={"ID":"f6f630f2-c9da-485c-b474-6a5eb53d70bd","Type":"ContainerDied","Data":"24efd222a72ac717b3505fcc393f3636e733b1c9dc79e6e91b9fd1f8fa87278d"}
Jul 28 12:30:03.190156 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:03.189730    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24efd222a72ac717b3505fcc393f3636e733b1c9dc79e6e91b9fd1f8fa87278d"
Jul 28 12:30:03.599011 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:03.598981    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-zwcm4_16bdac86-0c58-4ceb-891c-a9b36d22e6be/kuadrant-console-plugin/0.log"
Jul 28 12:30:03.868062 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:03.867994    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-jpcq4_9ea35308-e3df-4508-b643-50aa6aeb8ba8/manager/0.log"
Jul 28 12:30:03.991711 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:03.991686    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-6t8t4_426d4fc8-ca99-4c4d-bef0-e2aab6ab7afa/limitador/0.log"
Jul 28 12:30:06.105869 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:06.102281    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-4tm65_ca401824-eb63-4e42-a0b2-6d36572fc15d/main/0.log"
Jul 28 12:30:06.347148 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:06.347119    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-j88rq_cda6b35e-45c0-40f9-996c-1bf409ba9a27/main/0.log"
Jul 28 12:30:15.448450 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:15.448407    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fg8rp_1a04e1e5-57de-4b91-b28e-f4ef1b5e67be/global-pull-secret-syncer/0.log"
Jul 28 12:30:15.632093 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:15.632064    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xwtf2_0b0fd488-b3f4-4e45-8832-a74efd910839/konnectivity-agent/0.log"
Jul 28 12:30:15.689039 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:15.689014    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-155.ec2.internal_8a2ad931bbcb3d6172f2d426a64e2848/haproxy/0.log"
Jul 28 12:30:20.197073 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:20.197045    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-zwcm4_16bdac86-0c58-4ceb-891c-a9b36d22e6be/kuadrant-console-plugin/0.log"
Jul 28 12:30:20.409336 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:20.409304    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-jpcq4_9ea35308-e3df-4508-b643-50aa6aeb8ba8/manager/0.log"
Jul 28 12:30:20.437131 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:20.437111    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-6t8t4_426d4fc8-ca99-4c4d-bef0-e2aab6ab7afa/limitador/0.log"
Jul 28 12:30:22.678060 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:22.678038    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ssmwx_74056816-9582-42e1-a90c-fc0a94603c18/node-exporter/0.log"
Jul 28 12:30:22.702336 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:22.702313    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ssmwx_74056816-9582-42e1-a90c-fc0a94603c18/kube-rbac-proxy/0.log"
Jul 28 12:30:22.727945 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:22.727924    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ssmwx_74056816-9582-42e1-a90c-fc0a94603c18/init-textfile/0.log"
Jul 28 12:30:22.862407 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:22.862381    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9eb1e89f-7334-4d90-b463-6a99faf7230e/prometheus/0.log"
Jul 28 12:30:22.909389 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:22.909367    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9eb1e89f-7334-4d90-b463-6a99faf7230e/config-reloader/0.log"
Jul 28 12:30:22.935016 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:22.934966    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9eb1e89f-7334-4d90-b463-6a99faf7230e/thanos-sidecar/0.log"
Jul 28 12:30:22.962206 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:22.962186    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9eb1e89f-7334-4d90-b463-6a99faf7230e/kube-rbac-proxy-web/0.log"
Jul 28 12:30:22.987192 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:22.987160    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9eb1e89f-7334-4d90-b463-6a99faf7230e/kube-rbac-proxy/0.log"
Jul 28 12:30:23.013382 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:23.013348    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9eb1e89f-7334-4d90-b463-6a99faf7230e/kube-rbac-proxy-thanos/0.log"
Jul 28 12:30:23.040409 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:23.040393    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9eb1e89f-7334-4d90-b463-6a99faf7230e/init-config-reloader/0.log"
Jul 28 12:30:25.471921 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.471846    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"]
Jul 28 12:30:25.472278 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.472143    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6f630f2-c9da-485c-b474-6a5eb53d70bd" containerName="cleanup"
Jul 28 12:30:25.472278 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.472154    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6f630f2-c9da-485c-b474-6a5eb53d70bd" containerName="cleanup"
Jul 28 12:30:25.472278 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.472206    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6f630f2-c9da-485c-b474-6a5eb53d70bd" containerName="cleanup"
Jul 28 12:30:25.475532 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.475509    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"
Jul 28 12:30:25.477432 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.477410    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dwgdc\"/\"default-dockercfg-6rr66\""
Jul 28 12:30:25.477539 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.477417    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dwgdc\"/\"kube-root-ca.crt\""
Jul 28 12:30:25.477991 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.477974    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dwgdc\"/\"openshift-service-ca.crt\""
Jul 28 12:30:25.484785 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.484764    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"]
Jul 28 12:30:25.604885 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.604849    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e9b41f4-2c9e-40bd-a2f0-e842183cc353-lib-modules\") pod \"perf-node-gather-daemonset-vbmmc\" (UID: \"9e9b41f4-2c9e-40bd-a2f0-e842183cc353\") " pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"
Jul 28 12:30:25.604885 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.604887    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e9b41f4-2c9e-40bd-a2f0-e842183cc353-podres\") pod \"perf-node-gather-daemonset-vbmmc\" (UID: \"9e9b41f4-2c9e-40bd-a2f0-e842183cc353\") " pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"
Jul 28 12:30:25.605133 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.604915    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e9b41f4-2c9e-40bd-a2f0-e842183cc353-sys\") pod \"perf-node-gather-daemonset-vbmmc\" (UID: \"9e9b41f4-2c9e-40bd-a2f0-e842183cc353\") " pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"
Jul 28 12:30:25.605133 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.605010    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv55j\" (UniqueName: \"kubernetes.io/projected/9e9b41f4-2c9e-40bd-a2f0-e842183cc353-kube-api-access-pv55j\") pod \"perf-node-gather-daemonset-vbmmc\" (UID: \"9e9b41f4-2c9e-40bd-a2f0-e842183cc353\") " pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"
Jul 28 12:30:25.605133 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.605093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e9b41f4-2c9e-40bd-a2f0-e842183cc353-proc\") pod \"perf-node-gather-daemonset-vbmmc\" (UID: \"9e9b41f4-2c9e-40bd-a2f0-e842183cc353\") " pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"
Jul 28 12:30:25.706282 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.706252    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e9b41f4-2c9e-40bd-a2f0-e842183cc353-lib-modules\") pod \"perf-node-gather-daemonset-vbmmc\" (UID: \"9e9b41f4-2c9e-40bd-a2f0-e842183cc353\") " pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"
Jul 28 12:30:25.706282 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.706283    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e9b41f4-2c9e-40bd-a2f0-e842183cc353-podres\") pod \"perf-node-gather-daemonset-vbmmc\" (UID: \"9e9b41f4-2c9e-40bd-a2f0-e842183cc353\") " pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"
Jul 28 12:30:25.706502 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.706309    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e9b41f4-2c9e-40bd-a2f0-e842183cc353-sys\") pod \"perf-node-gather-daemonset-vbmmc\" (UID: \"9e9b41f4-2c9e-40bd-a2f0-e842183cc353\") " pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"
Jul 28 12:30:25.706502 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.706346    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv55j\" (UniqueName: \"kubernetes.io/projected/9e9b41f4-2c9e-40bd-a2f0-e842183cc353-kube-api-access-pv55j\") pod \"perf-node-gather-daemonset-vbmmc\" (UID: \"9e9b41f4-2c9e-40bd-a2f0-e842183cc353\") " pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"
Jul 28 12:30:25.706502 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.706396    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e9b41f4-2c9e-40bd-a2f0-e842183cc353-proc\") pod \"perf-node-gather-daemonset-vbmmc\" (UID: \"9e9b41f4-2c9e-40bd-a2f0-e842183cc353\") " pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"
Jul 28 12:30:25.706502 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.706430    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e9b41f4-2c9e-40bd-a2f0-e842183cc353-podres\") pod \"perf-node-gather-daemonset-vbmmc\" (UID: \"9e9b41f4-2c9e-40bd-a2f0-e842183cc353\") " pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"
Jul 28 12:30:25.706502 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.706445    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e9b41f4-2c9e-40bd-a2f0-e842183cc353-sys\") pod \"perf-node-gather-daemonset-vbmmc\" (UID: \"9e9b41f4-2c9e-40bd-a2f0-e842183cc353\") " pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"
Jul 28 12:30:25.706502 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.706430    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e9b41f4-2c9e-40bd-a2f0-e842183cc353-lib-modules\") pod \"perf-node-gather-daemonset-vbmmc\" (UID: \"9e9b41f4-2c9e-40bd-a2f0-e842183cc353\") " pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"
Jul 28 12:30:25.706502 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.706472    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e9b41f4-2c9e-40bd-a2f0-e842183cc353-proc\") pod \"perf-node-gather-daemonset-vbmmc\" (UID: \"9e9b41f4-2c9e-40bd-a2f0-e842183cc353\") " pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"
Jul 28 12:30:25.715209 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.715181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv55j\" (UniqueName: \"kubernetes.io/projected/9e9b41f4-2c9e-40bd-a2f0-e842183cc353-kube-api-access-pv55j\") pod \"perf-node-gather-daemonset-vbmmc\" (UID: \"9e9b41f4-2c9e-40bd-a2f0-e842183cc353\") " pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"
Jul 28 12:30:25.787176 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.787115    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"
Jul 28 12:30:25.914051 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:25.914026    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"]
Jul 28 12:30:25.916618 ip-10-0-141-155 kubenswrapper[2561]: W0728 12:30:25.916570    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9e9b41f4_2c9e_40bd_a2f0_e842183cc353.slice/crio-c01d7636a11681825df386c73ff187ecbe95d268218f2ba3d5b5c5bf781936b2 WatchSource:0}: Error finding container c01d7636a11681825df386c73ff187ecbe95d268218f2ba3d5b5c5bf781936b2: Status 404 returned error can't find the container with id c01d7636a11681825df386c73ff187ecbe95d268218f2ba3d5b5c5bf781936b2
Jul 28 12:30:26.293047 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:26.292994    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc" event={"ID":"9e9b41f4-2c9e-40bd-a2f0-e842183cc353","Type":"ContainerStarted","Data":"36cd9267a286afa6cab808bcc1bca3b2dbe6abbc30fde348e8018c3318d6cbf1"}
Jul 28 12:30:26.293271 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:26.293171    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"
Jul 28 12:30:26.293271 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:26.293229    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc" event={"ID":"9e9b41f4-2c9e-40bd-a2f0-e842183cc353","Type":"ContainerStarted","Data":"c01d7636a11681825df386c73ff187ecbe95d268218f2ba3d5b5c5bf781936b2"}
Jul 28 12:30:26.328928 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:26.328881    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc" podStartSLOduration=1.328866377 podStartE2EDuration="1.328866377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:30:26.328117429 +0000 UTC m=+3578.186773427" watchObservedRunningTime="2026-07-28 12:30:26.328866377 +0000 UTC m=+3578.187522377"
Jul 28 12:30:27.835646 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:27.835617    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f7m7q_f978d864-c551-4fcf-8a60-ef424a5556a0/dns/0.log"
Jul 28 12:30:27.859029 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:27.859004    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-f7m7q_f978d864-c551-4fcf-8a60-ef424a5556a0/kube-rbac-proxy/0.log"
Jul 28 12:30:27.907332 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:27.907310    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hpn2d_e552bdf3-9456-4e29-b6c8-aaad902d9d5c/dns-node-resolver/0.log"
Jul 28 12:30:28.481053 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:28.481025    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5c46895c78-mnn2v_0b4b1d25-a678-4bfa-8320-d0210806a661/registry/0.log"
Jul 28 12:30:28.532826 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:28.532795    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pthzf_9af318d1-7622-4574-8c01-ee23cbcf4fdc/node-ca/0.log"
Jul 28 12:30:30.791207 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:30.791179    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x4jk8_ee38fb9c-adce-4aae-9896-5b124fac12bf/serve-healthcheck-canary/0.log"
Jul 28 12:30:31.403109 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:31.403080    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-88547_34712309-48b9-4e19-a7dc-92be37026a55/kube-rbac-proxy/0.log"
Jul 28 12:30:31.428838 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:31.428816    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-88547_34712309-48b9-4e19-a7dc-92be37026a55/exporter/0.log"
Jul 28 12:30:31.453897 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:31.453879    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-88547_34712309-48b9-4e19-a7dc-92be37026a55/extractor/0.log"
Jul 28 12:30:32.307285 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:32.307250    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dwgdc/perf-node-gather-daemonset-vbmmc"
Jul 28 12:30:33.803563 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:33.803533    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-l6wtq_d7032b5a-b6d6-432a-9608-41553f3bf317/manager/0.log"
Jul 28 12:30:34.328526 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:34.328492    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6755d7577-8fmcq_463fa1da-5fff-4de1-815b-2f76070d5165/manager/0.log"
Jul 28 12:30:34.355265 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:34.355242    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6755d7577-8fmcq_463fa1da-5fff-4de1-815b-2f76070d5165/copy-manifests/0.log"
Jul 28 12:30:36.004302 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:36.004252    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-65bf6bb88c-rjm4f_f09f2213-fe3e-42f1-ad6f-e750825486da/manager/0.log"
Jul 28 12:30:42.937123 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:42.937091    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wprpw_bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56/kube-multus-additional-cni-plugins/0.log"
Jul 28 12:30:42.964558 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:42.964537    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wprpw_bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56/egress-router-binary-copy/0.log"
Jul 28 12:30:42.997396 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:42.997375    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wprpw_bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56/cni-plugins/0.log"
Jul 28 12:30:43.023609 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:43.023572    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wprpw_bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56/bond-cni-plugin/0.log"
Jul 28 12:30:43.049709 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:43.049693    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wprpw_bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56/routeoverride-cni/0.log"
Jul 28 12:30:43.075962 ip-10-0-141-155 kubenswrapper[2561]: I0728 12:30:43.075941    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wprpw_bf0d1c3d-b6b1-4a4a-baac-bdddc95a5b56/whereabouts-cni-bincopy/0.log"